That's a fair point, using 'interpreter' specifically was imprecise language on my part. My main point was php-fpm is developed by the core PHP team and is often the default in how PHP projects deploy these days, and that CVE was very similar to the recent 'fail' LPE vulnerabilities in the kernel.
My first reaction was, "watch, they're going to replace actual rigorous educational institutions with religious colleges" and lo and behold, "Liberty University" is at the top of the list for replacement civilian institutions.
"One of these things is not like the other, one of these things just doesn't belong. Find which thing is not like the other by the time I finish this song!"
From the description I thought the expression was a function of only 't', and there was no (for instance) accumulation of the previously computed byte. Then in the image I saw the same value of 't' evaluating to different values:
whether 'someField' is volatile or not. The volatile just affects the load/store semantics of the GETFIELD/PUTFIELD ops. For atomic increment you have to go through something like AtomicInteger that will internally use an Unsafe instance to ensure it emits a platform-specific atomic increment instruction.
We're kinda splitting hairs here. The significance, to me, is that the bolt does not have a very compelling drag coefficient compared to e.g. the Tesla M3 or the Ioniq 6.
Wtf are you talking about? I said budget sedan. Bolt is not a sedan. It doesn't compete with vehicles like the Tesla Model 3 and Hyundai Ioniq 6, which have much lower drag coefficients and consequently significantly longer range despite comparable battery sizes.
I love my Bolt, but it's not a sedan. That said, it's not terribly surprising GM is not interested in making an electric sedan given that it's been declining in popularity over time. Ford isn't even making any sedans at all apart from the Mustang.
It's too bad the 2023 Bolts are for the most part unavailable new, even though the used 2023 bolts have super low miles, and thus not qualified for the IRA rebate.
Postgres is great as a queue, but this post doesn't really get into the features that differentiate it from just polling, say SQL Server for tasks.
For me, the best features are:
* use LISTEN to be notified of rows that have changed that the backend needs to take action on (so you're not actively polling for new work)
* use NOTIFY from a trigger so all you need to do is INSERT/UPDATE a table to send an event to listeners
* you can select using SKIP LOCKED (as the article points out)
* you can use partial indexes to efficiently select rows in a particular state
So when a backend worker wakes up, it can:
* LISTEN for changes to the active working set it cares about
* "select all things in status 'X'" (using a partial index predicate, so it's not churning through low cardinality 'active' statuses)
* atomically update the status to 'processing' (using SKIP LOCKED to avoid contention/lock escalation)
* do the work
* update to a new status (which another worker may trigger on)
So you end up with a pretty decent state machine where each worker is responsible for transitioning units of work from status X to status Y, and it's getting that from the source of truth. You also usually want to have some sort of a per-task 'lease_expire' column so if a worker fails/goes away, other workers will pick up their task when they periodically scan for work.
This works for millions of units of work an hour with a moderately spec'd database server, and if the alternative is setting up SQS/SNS/ActiveMQ/etc and then _still_ having to track status in the database/manage a dead-letter-queue, etc -- it's not a hard choice at all.
I haven’t had the opportunity to use it in production yet - but it’s worth keeping in mind.
I’ve helped fix poor attempts of “table as queue” before - once you get the locking hints right, polling performs well enough for small volumes - from your list above, the only thing I can’t recall there being in sql server is a LISTEN - but I’m not really an expert on it.
Came here to mention Service Broker. I've used it in production in multi-server configurations for a number of years. It works really well but it's terribly obscure. Nobody seems to know it's even there.
The learning curve is steep and there are some easy anti-patterns you can fall into. Once you grok it, though, it really is very good.
The LISTEN functionality is absolutely there. Your activation procedure is invoked by the server upon receipt of records into the queue. It's very slick. No polling at all.
> * use LISTEN to be notified of rows that have changed that the backend needs to take action on (so you're not actively polling for new work)
> * use NOTIFY from a trigger so all you need to do is INSERT/UPDATE a table to send an event to listeners
Could you explain how that is better than just setting up Event Notifications inside a trigger in SQL Server? Or for that matter just using the Event Notifications system as a queue.
The argument of the OOP I was responding to was about how Postgres was better than other SQL solutions due to 4 reasons, with SQL Server being explicitly named. I was merely wondering whether his reasoning actually considered the abilities of SQL Server.
Admittedly I used SQL Server pretty heavily in the mid-to-late-2000s but haven't kept up with it in recent years so my dig may have been a little unfair.
Agree on READPAST being similar to SKIP_LOCKED, and filtered indexes are equivalent to partial indexes (I remember filtered indexes being in SQL Server 2008 when I used it).
Reading through the docs on Event Notifications they seem to be a little heavier and have different deliver semantics. Correct me if I'm wrong, but Event Notifications seem to be more similar to a consumable queue (where a consumer calling RECEIVE removes events in the queue), whereas LISTEN/NOTIFY is more pubsub, where every client LISTENing to a channel gets every NOTIFY message.
Using the INSERT/UPDATES is kind of limiting for your events. Usually you will want richer event (higher level information) than the raw structure of a single table. Use this feature very sparingly. Keep in mind that LISTEN should also ONLY be used to reduce the active polling, it is not a failsafe delivery system, and you will not get notified of things that happened while you were gone.
For my use cases the aim is really to not deal with events, but deal with the rows in the tables themselves.
Say you have a `thing` table, and backend workers that know how to process a `thing` in status 'new', put it in status 'pending' while it's being worked on, and when it's done put it in status 'active'.
The only thing the backend needs to know is "thing id:7 is now in status:'new'", and it knows what to do from there.
The way I generally build the backends, the first thing they do is LISTEN to the relevant channels they care about, then they can query/build whatever understanding they need for the current state. If the connection drops for whatever reason, you have to start from scratch with the new connection (LISTEN, rebuild state, etc).
I use a generic subsystem modeled loosely after SQS and Golang River.
I have a visible_at field which indicates when the "message" will show up in checkout commands. When checked out or during a heartbeat from the worker this gets bumped up by a certain amount of time.
When a message is checked out, or re-checked out, a key(GUID) is generated and assigned. To delete the message this key must match.
A message can be checked out if it exists and the visible_at field is older or equal to NOW.
That's about it for semantics. Any further complexity, such as workflows and states, are modeled in higher level services.
If I felt it mattered for perf and was worth the effort I might model this in a more append-only fashion taking advantage of HOT updates and etc. Maybe partition the table by day and drop partitions older than longest supported process. Use the sparse index to indicate deleted.. Hard to say though with SSDs, HOT, and the new btree anti-split features..
Thanks for the comprehensive reply, does the following argument stand up at all? (Going on the assumption that LISTEN is one more concept and one less concept is a good thing).
If I have say 50 workers polling the db, either it’s quiet and there's no tasks to do - in which case I don't particularly care about the polling load. Or, it's busy and when they query for work, there's always a task ready to process - in this case the LISTEN is constantly pinging, which is equivalent to constantly polling and finding work.
Regardless, is there a resource (blog or otherwise) you'd reccomend for integrating LISTEN with the backend?
In a large application you may have dozens of tables that different backends may be operating on. Each worker pool polling on tables it may be interested on every couple seconds can add up, and it's really not necessary.
Another factor is polling frequency and processing latency. All things equal, the delay from when a new task lands in a table to the time a backend is working on it should be as small as possible. Single digit milliseconds, ideally.
A NOTIFY event is sent from the server-side as the transaction commits, and you can have a thread blocking waiting on that message to process it as soon as it arrives on the worker side.
So with NOTIFY you reduce polling load and also reduce latency. The only time you need to actually query for tasks is to take over any expired leases, and since there is a 'lease_expire' column you know when that's going to happen so you don't have to continually check in.
Usual way is you update the table with a timestamp when the task was taken. Have one periodic job which queries the table looking for tasks that have outlived the maximum allowed processing time and reset the status so the task is available to be requeued.
The golden rule is "do not rewrite history of a public branch". Rebase/squash your PR branches to your heart's content, but once it's merged that's it.
You get clean history by not merging branches with 50 intermediary "fiddling with X" commits in them.
Related, it is a bad idea to use long-running per team branches. Merge early, merge often, get commits to trunk ASAP. With best practices on unit testing, code review, and so on, this scales to many thousands of developers. And will save a lot of pain over time.
(note, the broken tests were broken by the changes in the PR)
As soon as that PR is merged those commits are worthless. And there are branches with dozens of those "fixing X" commits that would otherwise pollute the commit graph.
Things like this should not be standalone commits though, they should be incorporated into the previous branch by amending the original work. It takes some effort to have a useful git history, it does not just happen on its own.
You can very easily rewrite your commit message on GitHub when squash merging. Since the organizations I work exclusively use squash merge, I often just update the commit to be more valuable, listing the important changes it contains. (And of course the PR in GitHub will contain the commit history of the branch that was squashed, as well as any discussion.)
IMO, this is a lot simpler and easier to do than rebasing your branch to have a flawless history.
Yep, intermediate commits on a branch tend to be completely worthless. I'd much rather have "git blame" point to the commit that contains the entire change together.
> I die a little bit each time I try to understand what changes were related to a line when tracking down a bug
A change/feature/bug is a branch, which is squashed into a commit on your main branch, right?
So your main branch should be a linear history of changes, one change per commit.
Because unless it's the most trivial of features, you'll break it up into smaller commits which each explain what they are doing and make reviewing the change easier.
As a simple example, I recently needed to update a json document that was a list of objects. I needed to add a new key/value to each object. The document had been hand edited over the years and had never been auto-formatted. My PR ended up being three commits:
1. Reformat the document with jq. Commit title explains it's a simple reformat of the document and that the next commit will add `.git-blame-ignore-revs` so that the history of the document isn't lost in `git blame` view.
2. Add `.git-blame-ignore-revs` with the commit ID of (1).
3. Finally, add the new key/value to each object.
The PR then explains that a new key/value has been added, mentions that the document was reformatted through `jq` as part of the work, a recommends that the reviewer step through the commits to ignore the mechanical change made by (1).
A followup PR added a pre-commit CI step to keep the document properly linted in the future.
In general I agree with you, there are absolutely times where you want to retain commit history on a particular branch (although I try to keep the source tree from knowing about things like commit IDs).
I would argue that those are by far the minority of PRs that I see. As I mentioned in another comment, _most_ PRs that I see have a ton of intermediary commits that are only useful for that branch/PR/review process (fixing tests, whitespace, etc). Generally the advice I give teams is, "squash by default" and then figure out where the exceptions to that rule are. That's mainly because, in my opinion, the downsides of a noisy commit graph filled with "addressing review comments" (or whatever) commits are a much bigger/frequent issue than the benefits you talk about. It really depends on the team.
> As I mentioned in another comment, _most_ PRs that I see have a ton of intermediary commits that are only useful for that branch/PR/review process (fixing tests, whitespace, etc).
Right, but that's only because developers don't amend and force push their commits to the PR branch as they receive feedback. Which is largely encouraged by GitHub being a terrible code review tool.
To me, git is part of the development process, it's not an extra layer of friction on top. So I compose my commits as I go. I find it helpful for recording what I'm thinking as I write the code. If I wait till the very end, I'll have forgotten some important bit of context I wanted to include. So during the day I may use the commits like save points. But before I push anything I'll often check out a new branch and create and incremental set of commits that have the change broken down into digestible pieces. And if I receive feedback, I'll usually amend those changes into the PR and force push it.
I'd like to add that I spend a lot of time cleaning up tech debt. And I deal with a ton of commits and PRs that don't explain themselves. So I'm really biased toward a clean development workflow because I hope to make the lives of those who come after me easier.
I was also trained on this workflow by being an early git contributor and it had extremely high standards for documenting its work. There's a commit from Jeff King that's a one line change with about six paragraphs of explanation.
There's no right answer here. I value the "meta" part of writing code. Not everyone does and that's okay.
> only useful for that branch/PR/review process (fixing tests, whitespace, etc).
I have had bugfix cases where, digging through the repo history, both of those examples accidentally introduced the bug (the first because the person who made the original change didn't completely understand a business rule so it changed both the code and the test, the second because of a typo in python that only affected a small subset of the data). Keeping the commit separate let me see very quickly what happened and what the intent actually was.
Because now instead of having a line changed within a granular level of changes, it's lost with the other changes from the same feature branch, which is a more macro level. So if a change in config is needed for the feature, the part when this config change actually need to be handled, or would impact the data-flow is harder to evaluate now that you mix it with template changes, style changes, new interactions needed for the users, etc...
EDIT: On top of that, there's usually a bit of 'related' work you need for a task, by example when you find an edge case related to your feature, and now you also needed to fix a bug, or you did a bit of refactoring on a related service, or needed to change the data on a badly formatted JSON file.
Unbeknownst to you, you added a bug when refactoring the related service, a bug that is spotted a few months after, only on a very specific edge case. If the cause is not obvious, you might want to reach for git bisect, but that won't be very useful now that everything I've talked about is squashed into a single commit.
> EDIT: On top of that, there's usually a bit of 'related' work you need for a task, by example when you find an edge case related to your feature, and now you also needed to fix a bug, or you did a bit of refactoring on a related service, or needed to change the data on a badly formatted JSON file.
I agree that's related work, but I'd argue that work doesn't belong in that branch. If you find a bug in the process of implementing a feature, create a bugfix branch that is merged separately. If you need to refactor a service, that's also a separate branch/PR.
That's actually the most common pushback I get from people when I talk about squashing. They say "but then a bunch of unrelated changes will be lumped together in the same commit", to which I respond, "why are a bunch of unrelated changes in the same branch/PR?"
I agree with you in principle, but it's usually because of process and friction. In the place I'm working right now, that would result in days lost as I need to create a new Jira ticket, which obviously require a team meeting for grooming (because Agile!), and then going after colleagues so that the PR is accepted, which best case still need for CI/CD pipeline to finally deploy, and then merge it to the dev branch, and finally rebase the current feature branch... and all this multiple times.
Because sometimes a PR touches more code than a single commit, and you lose the more granular context surrounding the more granular changes. You can always ask git to make the log more coarse, but once you “destroy” the granular history it is for all intents and purposes gone.
https://www.sentinelone.com/vulnerability-database/cve-2021-...