Stupid questions mostly regarding dehacked (especially directions, turning enemies to decos etc)

I'm not an expert on Vanilla limitations and I can't answer everything, but I might be able to answer 1 or 2 things as well as point out some resources you might want to bookmark.

About changing enemies into decorations: Yes, you can turn enemies into decorations by changing its various attributes (health, speed, bit field attributes, etc.), removing most of its state designations and changing its spawn states. You can also turn decorations into enemies (you need to give them the same type of states and attributes that other monsters tend to have). You can also have destructible decorations (you can model them after the explosive barrel, but without the explosion action itself). One thing to note however: if you turn an enemy into a decoration, you will break the ending roll call screen. Sometimes even custom enemies break that screen (The wad Strain has a floating turret that dies in a unique way and its death state causes the roll call to break).

About the multidirectional sprites: Yes, you can turn multidirectional sprites into single direction sprites. You can also take one directional sprites and give them 8 directions. I have an example wad that has a shotgunner that moves with one directional sprites and a candle that has 8 directions (2 unique directions and 3 pairs that are mirrored), I colored them both in weird ways to make it more apparent.
The test wad (it does work in my version of Chocolate Doom, but it needs to be launched with the -merge parameter instead of the -file parameter): testsprite.zip

As for the other stuff, here are some useful resources that you might be able to use:

• The Doomwiki page on DeHackEd - https://doomwiki.org/wiki/DeHackE
• Some of Doomkid's tutorials  
  Spoiler

   

   

• The "boomdeh.txt" file from the Boom source port (it's made for Boom, but it does have a lists all the states from Vanilla Doom, it also lists which states preform which actions) - https://github.com/fragglet/mbf/blob/master/boomdeh.txt
• The "frames.edf" file from the Eternity Engine source port (same as the "boomdeh.txt" file, but it might be easier to read, because it actually tells you which states belong to which entities, you might want to use this one together with the "boomdeh.txt" file) - https://github.com/team-eternity/eternity/blob/master/base/doom/frames.edf
• "Naming conventions for Doom Sprites" by Enjay - https://aspectsweb.co.uk/dehacked/doomsprites1.html
• "Adding and Replacing Doom Sprites" by Enjay - https://aspectsweb.co.uk/dehacked/doomsprites2.html
• Enjay's table of Doom sprite names (orginised by Dehacked frame number) - https://aspectsweb.co.uk/dehacked/doomsprites3.html
• Enjay's table of Doom sprite names (orginised by wad sprite name) - https://aspectsweb.co.uk/dehacked/doomsprites4.html
• The Unofficial Doom Specs (Chapter 5 might be of some note) - https://www.gamers.org/dhs/helpdocs/dmsp1666.html
• The Doom sound table from the Eternity Wiki (everything after 108 is non-vanilla) - https://eternity.youfailit.net/wiki/DeHackEd_/_BEX_Reference/Sound_Table
• The REKKR wad and .deh patch (One of the best resources I've found has honestly been other wads. The REKKR wad has replaced sprites that in the original Doom had one direction, but have multiple in the REKKR wad, as well as one monster that has one directional sprites and the .deh patch has  destructible decorations. You can pop them into SLADE or Whacked4 to explore them further) - https://www.moddb.com/mods/rekkr

On 12/26/2023 at 7:17 AM, poisonhead said:

Also for some reason my .deh got corrupted or something, a few items had changed back to defaults but I guess that's my stupidity which reminded me, if I'm using Boom, do I put .bex or .deh into a wad or both? So far been only using .deh but in Whacked4 I selected Boom format even though I haven't used any Boom specific things except likely exceeded a few string lengths, what happens then to the .deh? Are they cut, included, reverted to default or do they cause problems? Should I selected Doom format instead of Boom if I'm not using Boom stuff? (just mapping for Boom made me automatically select Boom and I'd like the longer strings). Are .deh files always vanilla Doom format or do Boom .deh files have stuff Vanilla don't like? Which brings me back to my question that if I use longer strings etc should I include .deh, .bex or both and how would I go about the latter? I know dehacked lump but is there some lump name for .bex?

Maybe your .deh got corrupted because you wrote some strings that were longer than they were in Vanilla Doom and Whacked4 changed your patch to the Boom format as a fallback mechanism (although, I don't really know that much about Whacked4, I used it a few years ago, but now I tend to write my dehacked patches in the .bex format by hand).

Also, the file extension for the dehacked patch file does not determine what compatibility the patch has (with one exception). A .deh patch file can actually contain Boom, MBF, or even MBF21 features. The main difference between a .deh and .bex file is that .deh files store most data with numeric values and .bex has data that is more human readable. The main reason to use the .deh format is in conjunction with the DOS program "DeHackEd" to create a copy of the Vanilla Doom executable that has all the changes from the patch applied into the copy (The Doomkid videos have examples of this). The .bex formatted files are meant for source ports only, but all source ports can also read .deh patches.

The Dehacked lump inside of a wad file can be written in the .deh format or the .bex format. In other words: if the name of the inner Dehacked lump is "Dehacked" and you paste the contents of a .deh patch file - it will work in most if not all source ports, and if the name of the inner Dehacked lump is "Dehacked" and you paste the contents of a .bex patch file - it will still work in most source ports, but a few might not work (it definitely will not work in Chocolate Doom and it might not work in Crispy Doom, but I'm not sure about the last one).

If you have to use longer strings, you might as well place your dehacked patch into the wad itself (as stated previously, either of the 2 formats will work), but at that point - the wad won't be Vanilla compatible due to the fact that the strings are longer than they were in the original. If you manage to make a dehacked patch that does have strings that are as long as they were in Vanilla - you could keep the .wad and .deh file as separate files, however, you could also add a copy of the outer the .deh patch as an inner dehacked lump into the wad (a lot of modern Vanilla wads tend to do this - have a .deh file and a .wad file, but the .wad file also contains a inner copy of the .deh file as a lump).

Sorry if this response (as well as the previous one) is a bit too hectic and/or simplistic.

On 12/26/2023 at 8:17 AM, poisonhead said:

Are .deh files always vanilla Doom format or do Boom .deh files have stuff Vanilla don't like?

Bex absolutely has stuff that vanilla doom or rather, specifically, dehacked.exe for DOS, will not be able to understand. For example, consider vanilla codepointer editing (source: 1000 lines 3):

Pointer 1 (Frame 2)
Codep Frame = 174

Pointer 2 (Frame 3)
Codep Frame = 176

Pointer 3 (Frame 4)
Codep Frame = 783

Pointer 4 (Frame 6)
Codep Frame = 127

Pointer 5 (Frame 9)
Codep Frame = 157

Pointer 6 (Frame 10)
Codep Frame = 157

vs bex [CODEPTR] section (source: Eviternity II):

[CODEPTR]
FRAME 6 = WeaponMeleeAttack
FRAME 667 = Spawn
FRAME 1089 = Look
FRAME 1090 = Look
FRAME 1091 = Chase
FRAME 1092 = Chase
FRAME 1093 = Chase
FRAME 1094 = Chase
FRAME 1095 = Chase
FRAME 1096 = Chase
FRAME 1097 = Chase
FRAME 1098 = Chase

String editing is also completely different. Vanilla has Text notations for original strings in the exe (source: 1000 lines 3):

Text 7 7
SLIME16ZINTBG1
Text 7 7
RROCK14ZINTBG2
Text 7 7
RROCK07ZINTBG3
Text 7 7
RROCK17ZINTBG4
Text 7 7
RROCK13ZINTBG5
Text 7 7
RROCK19ZINTBG6
Text 17 13
HEAVY WEAPON DUDEPLASMA ZOMBIE
Text 43 37
You need a blue key to activate this objectYou need a blue key to activate this.
Text 42 39
You need a red key to activate this objectYou need a purple key to activate this.
Text 45 38
You need a yellow key to activate this objectYou need a green key to activate this.
Text 37 34

whereas bex adds [STRINGS] section with a much more limited set of mnemonics (source: Wormwood 5):

[STRINGS]
HUSTR_1 = map01: The Boiler
HUSTR_2 = map02: Spiral Castle
HUSTR_3 = map03: After Image
HUSTR_4 = -
HUSTR_16 = -
HUSTR_31 = map31: I'm Going to Kill You, With an Axe
HUSTR_32 = map32: Intense Ultra Slaughter
GOTBERSERK = Axe powered up!
GOTCLIP = Picked up a plasma cell.
GOTCLIPBOX = Picked up a plasma cell pack.
GOTCELL = Picked up a BFG cell.
GOTCELLBOX = Picked up a BFG cell pack.
GOTBLUECARD = Picked up a green keycard.
GOTBLUESKUL = Picked up a green skull.
GOTREDCARD = Picked up a red keycard.
GOTREDSKULL = Picked up a red skull.
GOTYELWCARD = Picked up an orange keycard.
GOTYELWSKUL = Picked up an orange skull.

And of course there is other stuff (see boomdeh.txt).

Using bex strings editing in Whacked4 won't allow you to change sprite names, which afaik is needed for vanilla exe with custom sprites. So you should probably take a pause, study vanilla limitations, practice relevant editing techniques (i.e. make a single custom sprite decoration show up in one-square-room.wad in dosbox) and make an informed decision as to whether you actually want to be vanilla-compatible.

If yes, then that's something you need to target from ground up - "I'll make boom stuff and then edit it to fit vanilla" will never work.

If no, then the most sane dehacked option in soon to be 2024 for a wad with a lot of custom decorations is MBF21/DSDHacked - forget the concept of "replacing" and "sacrificing" anything at all and just make new things/frames/sounds/sprites.

I still find it baffling that I cannot find a guide or a simple example of how to do vanilla ambient sounds. I've got an invisible dehacked thingy, what is the minimal thing needed to play a sound when it sees (or hears) the player? Is spawn state and alert sound enough? (and does the sound play more than once?)

Also what about looping sound, I remember seeing something about it somewhere but can't remember and if it was for vanilla.

I also wonder about replacing sprites, can I use enemies 0 direction (towards the player) frames for things? When I first tried trial and testing I quickly found out what the numbers meants (thing could be seen only from certain direction and turned into another thing etc). Nowadays my time is _limited_ and CPU fan gave out, even running light utilities (PSP7, XWE, Slade, Whacked, Goldwave) may cause this to turn off due to overheating so I'm a bit stuck trying to learn "blindly". Also when I tried to replace archviles flame animation with different things it didn't work out, is this a hard coded thing? I wish there was some dehacked guide that actually answers these kind of questions I've seen before with no straight answer. What enemy sprites can be safely turned into one frame decorative things? and think I may be able google or see in Whacked this last thing but is there a way to trigger archvile's flame animation easily?