Getting the difficulty of an encounter right is an important part of running a table top role-playing game. However, it is more of an art than a science and there are many options at the hands of the Dungeon Master. I hope to help newer Dungeon Masters by going over this topic at a higher level.
Difficulty is Relative
What makes a particular encounter difficult for a party comes from a bunch of different factors. The experience of the players has quite a bit to do with it. On top of this, equipment, initiative (in D&D like systems where players act in turns) and party composition also play very big roles. This can work in both directions. An encounter that may be deadly for a party of new players could be easily beaten by a party of experiences players by exploiting the weaknesses of the creatures and tactical choices in the terrain. As always, knowing your players and what they like is important. Difficulty for combat is also generally better details in rule system than for puzzles and other things. The difficulty in solving a riddle will largely depend on the knowledge of your players (though hints from good insight checks in D&D 5th edition can help too).
3 Basic Difficulties of Encounters
As a general idea, there are really 3 different difficulties that exist if the goal is to kill every enemy in the room. The encounter can be perfect for the party. Maybe a character or two will be making death saves by the end, but the party will probably win if they play reasonably and have luck somewhat on their side (a string of 1's can doom anyone). If the party hasn't rested for a while, the encounter can be more difficult than anticipated. In such a situation, the party may need to retreat.
The encounter could be easy for the party. They may take a couple of scratches but in general they aren't in too much danger at that moment. The danger from these kinds of encounters comes from something other than the encounter itself. Too many in a row force the party to expend resources that they may need for the more challenging part later. The possibility for the party to rest and recuperate still exists and will make these kinds of encounters even easier. For this kind of starving strategy, there needs to be a risk or loss associated with rest (otherwise they will just rest and make the encounter even easier). It can also significantly slow down the players in a situation where time is the most valuable resource they have (in such a case, resting will cost them time as well).
The encounter can also be extremely challenging for the party. Throwing a lich at a level 1 party is not just challenging but almost impossible for the party to win in terms of a combat rules perspective (you can still try to introduce a big bad this early and have the party run as allies are getting killed). In such a case, short of pure ingenuity and the Dungeon Master playing along (dropping the entire tunnel on the lich), the party doesn't have any hope of winning. They may need to run away (your players will need to know that running away is an acceptable choice). There is another kind of challenging the party. It is probably more desirable that the encounter will be won by the slimmest of margins unless the party plays smart and has luck on their side. This also means that the party could be killed by bad luck and that they may need to consider escaping to fight another day. This kind of encounter tends to be the ending of quest. Most of the time, they don't just stumble upon this kind of encounter while shopping for groceries.
The difficulty of an encounter plays directly into the story that is being told. When trying to create an immersive game, the kinds of encounters and when they occur will be important to the tone and the story itself. Your final fight doesn't need to challenge the players at full strength if it doesn't make sense. It may even be jarring if the physically weak but manipulative bad guy is able to hold himself as an equal to the fighter in sword fighting (if the idea is that the bad guy was looking for a way to become stronger and succeeded, sure, but in that case you have built an in world reason to justify it). If the hardest encounter you ever had involved a fistfight in a bar, it may also be a bit jarring for the party (once again, know your players since some may enjoy that situation from a comedic perspective).
Goals Change the Difficulty
The players' goals can have a big effective on the encounter itself. If the players find themselves in a room full of zombies and their goal is to get to the next room, they can choose to run past them and close the door behind them. Such a choice may make the combat encounter easier even if they later open the door (fighting them after choosing beneficial positions and after choke pointing them may be a lot easier) and in the meantime may allow them to completely bypass the encounter. It can also make things worse if there are more zombies in the new room and the zombies from the other room start to break down the door.
If the players are trying to assassinate someone, they don't have to kill everyone in the room. This makes the encounter easier from their perceptive (they will also probably get to choose the terms they attack under) but still dangerous. Even if the players can't beat everyone in the room, they can still take out their target and escape. Such a tactic is perfectly reasonable in such a case, especially if the players realize how hard it would be to kill everyone in the room.
There are other ways of getting past a combat encounter (giving a counterfeit of the item the bad guys want is a classic) and depending on the choice taken, there could be consequences later (the bad guys who got the counterfeit item you threw at them before you ran will come back and not listen to what you have to say, even if you offer them the real item).
Goals can also make an encounter more difficult for the party. If a group of cultists is trying to summon a demon and they are just 2 turns away from doing so when the party enters, the party's goal and the situation makes it more difficult than just killing cultists (of course, doing before the demon is summoned will work but may require some creativity). There is a chance the party will be able to stop the demon from being summoned (maybe by using burning hands on the alter, throwing the offering off of the alter or something else entirely) but if the demon does get summoned, they have an incentive to stay and fight (they won't be able to accomplish their goal if they run and the demon is still on this plane). They can still escape if things go very badly, but they will then need to take more time to finish off the quest and it may be even harder now that they are expected.
Not Always Combat
An encounter can be more than just combat. Traps and puzzles can be mixed together within a combat encounter to make things both easier and more difficult (you can make zombies run into traps, after all). The wizard may be able to lift a curse to free all of the ghosts attacking the party. If there is no combat but a riddle or puzzle, the challenge comes from trying to solve it and may take a long amount of in-game time. There may be easy ways for the players to talk their way out of otherwise impossible to win situations as well. There are many different options in such cases but they can still add, remove or have a difficulty of their own.