Well, try. Paranoia isn't healthy (or realistic in most cases).
Story points are relative because people have proven forever that they really, really stink at estimating in absolute terms. They get it wrong almost every time. You think something will take a day; I think it will take a week. Who's right? Me, usually, but that's fodder for another article.
The point is that while you and I may disagree on absolute timeframes, we will almost never disagree about relative levels of effort, e.g. is it more difficult to walk across the room or across a city block? That's what story points are for.
Tying story points to time "crosses the streams" (absolute vs relative) and destroys the value of story points entirely. No wonder you're conflicted.
Your real issue here seems to be with fixed deadlines and the agile process of reality-based negotiation with product managers about what's possible in a given timeframe with a given number of resources.
After breaking a feature down into stories and estimating those stories, it's time for a reality-based discussion with the business -- good news (based on the team's historical velocity and the number of estimated points, we think we can deliver this feature by the desired date) or bad news (based on the team's historical velocity and the number of estimated points, we don't think we can deliver this feature by the desired date).
If the latter, scope needs to be reduced and/or the engineering team needs to find ways to do more in the available time.
But the issue isn't with story points or the processes that utilize them. If you use them correctly, they work well.