Does anyone have experience using Apache Iceberg, Hudi or DeltaLake for storing logs instead of Loki?

danluu.com mentioned this approach (or just, ‘big data’ systems) for traces and metrics iirc. Not sure if for logs too. Aren’t all of those relatively tabular? What would you be looking for from those tools to help with logs?

Monads but more importantly MonadTransformers so you can program in a legible fashion.

However, there's a lot of manual labour to stuff everything into a monad, and then extract it and pattern match when your libraries don't match your choice of control flow monad(s)!

This is where I'd prefer if compilers could come in.

Imagine being in the bowels of a DB lib, and realising that the function you just write might be well positioned to terminate the TCP connection that it's using to talk to the database with. Oh no: now you have to update the signature and every single call-site for its parent, and its parent, and...

Instead, it would be neat if the compiler could treat things you deem cross-cutting as a graph traversal problem instead; call a cancelable method and all callers are automatically cancelable. Decisions about whether to spawn a cancelable subtree, to 'protect' some execution or set a deadline is then written on an opt-in basis per function; all functions compose. The compiler can visualise the tree of cancellation (or hierachical loggers, or OT spans, or actors, or green fibers, or ...) and it can enforce the global invariant that the entry-point captures SIGINT (or sets up logging, or sets up a tracer, or ...).

So imagine the infrastructure of a monad transformer, but available per-function on an opt-in basis. If you write your function to have a cleanup on cancellation, or write logs around any asynchronous barrier, the fiddly details of stuffing the monad is done by the compiler and optionally visualised and explained in the IDE. Your code doesn't have to opt-in, so you can make each function very clean.

Yes, there's plenty of space for automation and advanced support from tooling. Hell, not every perspective is best viewed as plaintext; in particular, anything that looks like a directed graph fundamentally cannot be well-represented in plaintext at all without repeating nodes, breaking the 1:1 correspondence between a token and a thing represented by that token.

Still, I believe the core insight here is that we need different perspectives at different times. Using your example, most of the time I probably don't care whether the code is cancellable or not. Any mention of it is distracting noise to me. But other times - perhaps next day, or perhaps just five minutes later, I suddenly need to know whether the code is cancellable, and perhaps I need to explicitly opt out of it somewhere. It's highly likely that in those cases, I may not care about things like error handling logic and passing around session identifiers, and I would like that to disappear in those moments, etc.

And hell, I might need an overview of the which code is or isn't protected, and that would be best served by showing me an interactive DAG of functions that I can zoom around and expand/collapse, so that's another kind of perspective. Etc.

EDIT:

And then there's my favorite example: the unending holy war of "few fat functions" vs. "lots of tiny functions". Despite the endless streams of Tweets and articles arguing for either, there is no right choice here - there's no right trade-off you can make here up front, and can never be, because which one is more readable depends strictly on why you're reading it. E.g. lots of tiny functions reduce duplication and can introduce a language you can use to effectively think about some code at a higher level - but if there's a thorny bug in there I'm trying to fix, I want all of that shit inlined into one, big function, that I can step through sequentially, following the actual execution order.

It is my firm belief that the ability to inline and uninline code on the fly, for yourself, personally, without affecting the actual execution or the work of other developers, is one of the most important missing piece in our current tooling, and making it happen is a good first step towards abandoning The Current Paradigm that is now suffocating us all.

Second one would be, along with inlining, the ability to just give variables and parameters fixed values when reading, and have those values be displayed and propagated through the code - effectively doing a partial simulation of code execution. Being able to do it ad hoc, temporarily, would be a huge aid in quickly understanding what some code does.

Promises are so incredibly close to being a representation of work.

The OS has such sophisticated tools for process management, but inside a process there are so many subprocesses going on, & it feels like we are flailing about with poorly managed process like things. (Everyone except Erlang.)

I love how close zx comes to touching the sky here. It's a typescript library for running processes, as a tagged template function returning a promise. *const hello = $`sleep 3; echo hello world`. But the promise isnt just a "a future value", it is A ProcessPromise for interacting with the promise.

I so wish promises were just a little better. It feels like such a bizarre tragedy to me the "a promise is a future value" not a thing unto itself won the day in es6 / es2015, destroyed the possibility of a promise being more; zx has run into a significant number of ergonomic annoyances because this small world dogma.

How cool it would be to see this go further. I'd love for the language to show what promises if any this promise is awaiting! I long for that dependency graph of subprocesses to start to show itself, not just at compile time but for the runtime to be able to actively observe and manage the subprocesses within it at runtime. We keep building workflow engines, build robust userland that manage their own subprocesses, user user lands, but the language itself seems so close & yet so far from letting the simple promise become more a process, and that seems like a sad shame.

Too bad Apple hasn't released Apple AI at all; why bother with press releases featuring it? (EU)

Seems to be it's an ebb and flow of complexity; sometimes it increases like with sidecar proxies and their issues, like how they might listen on NICs differently, or how they cause grey failures, and sometimes it decreases like when the sidecar moves to the kernel and gets rewritten in eBPF and we can remove another ip filtering-firewall-chain. So right now it seems to be a bit of an ebb, as these things are becoming moduralised.

We ran Zalando Operator for Postgres in k8s for a year, until finally succumbing to its technical debt that leaks out from every bit of its software being.

After switching to the Chrunchy Data pg operator v5 on k8s, we've had close to zero problems - one or two times a year the log shipping / HA replication fails and we have to restart it, but it's really neat! I can *warmly* recommend it; it really is CloudSQL in K8S.

Are these side-channel based management technologies turns on even on MacBook laptops?

That is a great question. I would assume that they are because the Intel management technology is currently built in to ALL Intel chips for the past 10 years. It may be a good thing that Apple is looking at building their own ARM based Macs.

> callbacks aren't OOP, they are per definition functional programming

You can't just make things up on the spot.

I know very little about functional programming (so I should probably just stop and wait for a more knowledgable commenter, but here I go...) but even I know that functional programming doesn't mean "programming that involves functions".

It's a totally different programming paradigm that involves specifying a collection of facts relating function inputs to their outputs and letting the compiler/interpreter figure out how to turn that into a program. There are lots of ways that these sorts of program are different from the type of programming you're used to, including pattern matching, lazy evaluation, and, yes, passing around functions as values. But a particularly surprising one is that often the order of statements don't make any difference because the program is not just executed sequentially starting at the top and working its way down. OOP is subtype of imperative programming, which is the usual programming you're used to where you just write a series of instructions that are executed top-to-bottom (except for function calls and flow control like "if" and "for" but even there you're explicitly specifying what should be executed next).

Apologies if you knew all of that, but it seemed you like didn't because surely no one that really understands functional programming would confuse it with imperative programming involving some callbacks.

---

I think the parent commenter called you out on this even though it's not the main thrust of your argument because it's a really significant misuse of terminology and shows a lack of understanding of fundamental programming concepts. Pedantically it's a bit of an ad hominem attack, but you seem to be coming down hard on Cap'n Proto especially because of its misuse of terminology and it's ironic that you're the one misusing it.

(Another example of this is where you object to "streaming", which means what the document say it does, which you call "polling" but that really means having to proactively check every so often whether something is done rather than getting a callback. [Edit: these are actually orthogonal concepts because even with a non-streaming request you could either be notified or have to poll for the single response. I think you have just totally missed what "streaming" means here.])

I am having to fight really hard the urge to dig into more detail of your comment. But I will leave it at one more thing that you seem to have missed: You seem to be talking as if this page is a first introduction to Cap'n Proto, when it's not, it's just a changelog entry intended for people that already know what the library is. Of course features are mentioned without a proper introduction, changelogs are typically of the form "add cancellation parameter to the floog() function" without explaining what "floog()" is. Adding all that detail would actually make them less useful, because it's just noise to the target audience that buries the real content, which is what's actually changed.

I do really appreciate that you try to explain this from first principles. First, to explain myself, I'm primarily programming in F#, which is a hybrid functional AND object oritented lang (also use: TS, Elm, Haskell, C#, Python, Ruby etc). I prefer to write the software in a functional style though, so I rather use closures and self-recusive functions acting as message loops over the dynamic dispatch of "telling" objects to do things.

> and, yes, passing around functions as values

If we go back to the foundations of what I would call functional programming, Lambda Calculus as defined by Church, I like to give the evaluation of booleans as a prime example of how functional programming computes by passing callbacks around: https://en.wikipedia.org/wiki/Lambda_calculus#Logic_and_pred...

Since this was arguably the very first functional programming "language" that existed in the world (even: one of the very ways to do structured computation!), and is almost exclusively uses callbacks, I'm very confident in saying that callbacks is more functional programming than anything else.

Functional programming still computes heavily via function composition; the principle of passing callbacks around is at the very center of functional programming. To name a few instances of callbacks being passed around; via currying "context" values, by passing a function to "act in a context" (hole-in-the-middle/delegator) or closing (closures) over mutable state to enable runtime re-configuration.

What most people don't realise about functional programming (until they've done it a long time) is that it's not in the syntactic features, nor in the type system, that functional programming really shines, it's in the compositionality of it.

> But a particularly surprising one is that often the order of statements don't make any difference

This is generally a false statement. You can mean many things here: dataflow programming (https://en.wikipedia.org/wiki/Dataflow_programming) or the ability of monadic computation expression to choose when in time it executes its built-up construction, or how lazy evaluation can share chunks (bits of compute) that are beta-(normalised)-normal to each other.

What is true, however, is that callbacks is not ONLY a core concept in functional programming (even if it's arguably MORE core in FP); even PL/1 and Cobol have address pointers that gets passed as callbacks, but there it's more used for cross-system interop than for composition. Assembly and even C also heavily use callbacks, but those are not OOP. The dynamic dispatch you have in object hierarchies in OOP, is not even such a close concept to callbacks (https://en.wikipedia.org/wiki/Dynamic_dispatch).

> write a series of instructions that are executed top-to-bottom

That's procedural programming, not OOP: https://en.wikipedia.org/wiki/Procedural_programming

---

> I think the parent commenter called you out on this even though it's not the main thrust of your argument because it's a really significant misuse of terminology and shows a lack of understanding of fundamental programming concepts. Pedantically it's a bit of an ad hominem attack, but you seem to be coming down hard on Cap'n Proto especially because of its misuse of terminology and it's ironic that you're the one misusing it.

Obviously I disagree with this (see above, with references!). The way he answered is unconstructive and only leads to hard feelings. I'm coming down hard on the way the Cap'n Proto article is written, and I can back it up and argue my point. I'm in no way attacking the author, or objectively saying that Cap'n Proto is bad — only that I think the way it's making its case is.

> Another example of this is where you object to "streaming"

I'm objecting not to "streaming" but to how the code makes it look. It's really unclear to someone who doesn't know the Cap'n Proto syntax. I think you're missing the point that my review is from someone who really understands these concepts, reading the article and forming an opinion of whether to give Cap'n Proto a chance based on it. Also, the article only says streaming means "returning multiple responses", which is an API-level concern, not a protocol concern (* * not a protocol concern _ necessarily _ * * : the protocol is what should be explained; for example gRPC uses HTTP2 push to do streaming responses)

> You seem to be talking as if this page is a first introduction to Cap'n Proto, when it's not, it's just a changelog entry intended for people that already know what the library is

It is the first introduction I had to Cap'n Proto, and I also after reading the article read the main site's introduction. I'm giving comments on how I perceive Cap'n Proto based on that.

> ... foundations of what I would call functional programming, Lambda Calculus as defined by Church ...

Ah! Now we’re getting to the nub of it. Yes, the lambda calculus, absolutely correct.

> ... almost exclusively uses callbacks ...

Whoops! This is absolutely wrong. The functions in the lambda calculus cannot have side effects (like saving a file or displaying a message to the user) and they can only depend on their inputs (e.g. they cannot return a number entered by a user, data from a socket, the current time, etc.). In the software and computer science world they are called "pure functions". Mathematicians (including Church) would simply call them "functions" because that, by definition, is what a mathematical function is (it is a subset of the Cartesian product of the domain and codomain, very much like a lookup table in computer programming except it can be infinite or even uncountably infinite).

These are very different from the callbacks being discussed in the original post. A callback - as the name suggests - is a function passed from a higher-level abstraction to a lower level abstraction to be notified when something happen. Those exist purely for their side effects - that's the whole point of a callback - and usually have no return value at all.

> Functional programming still computes heavily via function composition; the principle of passing callbacks around ... [e.g.] via currying "context" values, by passing a function to "act in a context" (hole-in-the-middle/delegator)

Sure, except you're passing (pure) functions around, not "callbacks" (at least for it to count as true functional programming).

> closing (closures) over mutable state

As you said, F# is a hybrid language, and mutable state is on the imperative side (admittedly a concession made even by stricter functional languages like LISP).

> This is generally a false statement. ... how lazy evaluation can share chunks (bits of compute)

It would be true in a pure functional language, and is only false to the extent that a real-world "functional" languages actually have some imperative features.

> > write a series of instructions that are executed top-to-bottom

> That's procedural programming, not OOP:

No, it's imperative programming, as I said in my previous comment. Procedural programming and OOP are both subtypes of imperative programming. On the Wikipedia page you linked to, look at the box on the right hand side, and note how imperative says "(contrast declarative)" next to it, and vice-versa. The box shows that declarative includes functional and dataflow programming.

> Assembly and even C also heavily use callbacks

Agreed, and they're a lot more similar to the type of callbacks being described in the original post than pure functions: they're totally imperative and allowed side effects. But they don't normally have included state - you usually get a void* for your extra data but no functions to copy and destroy that, unlike C++ lambdas and std::function (Python uses garbage collection to avoid needing those for state attached to function objects). So using them ends up with a slightly different feeling to the developer than callbacks in OOP languages, which is why the article specified callbacks in OOP languages rather than callbacks in general.

----

There are other details I could pick apart in your original comment or other replies. But there is a broader point. I know this isn't terribly constructive, but I think it's important to address the elephant in the room: frankly, none of your comments make any sense. You misuse terminology all over the place, but then pedantically pick apart individual technical terms in other people’s comments, mostly because they've used the terms with their real meanings rather than your imagined meanings. The commenter _pmf_ chose to highlight your incorrect criticism of "callbacks" and I followed up on it, but almost everything else in your original post deserves the same treatment; that was just one example. I really think people downvoted your original comment rather than replying because they took a look at it and thought "this is a lost cause".

In the comment I'm replying to, you say "I'm very confident in saying that" and complete the sentence with something that is wrong, and several people have already told you so. But this attitude comes across implicitly elsewhere. I implore you, please stop trying to just push your understanding of things, and instead lean back and consider that the comments contradicting you could actually be right.

This especially applies to looking back at Kenton's replies, even if you ignore what I'm saying (fair enough). I realise he gave up replying to you eventually but his replies were very reasonable and clear explanations and you responded with just an explosion of more confusion that would take more and more effort to clear up - I think he has the patience of a saint for addressing as much of your comments as he did.

Bear in mind that no one else in the comments had trouble with either the original article or the introduction to Cap'n Proto in general. No one came along and agreed with anything you said, which does happen when bad articles are posted here. In general (of course there are exceptions), Hacker News commenters are clever and reasonable. If you find yourself surrounded by clever reasonable people and they're all wrong and you alone are right, it is time to consider if it is really the other way round.

I ended up putting the wrong reply here to this frament, sorry about that:

> > This is generally a false statement. ... how lazy evaluation can share chunks (bits of compute)

> It would be true in a pure functional language, and is only false to the extent that a real-world "functional" languages actually have some imperative features.

What I had intended to say was that, yes, this is the sort of feature of functional languages I was talking about. In a true (perhaps theoretical) functional programming language, all functions are pure, so you can arbitrarily reorder function calls (relative to other calls of the same function and to calls to other functions) and this cannot possibly have an effect on the meaning of the program. You can even coalesce multiple calls to the same function with the same parameters.

As I'm sure you realise, this can't work for functions that have side effects e.g. reading bytes from a socket. Again, these are things that can happen for functions, including callbacks, in imperative programming languages (or the imperative bit of mixed or mostly-functional programming langauges).

Wow ok... lots of incorrect assumptions here. Just for fun let's address some of them.

> but that has a name, it's called polling, not streaming.

It's not polling. The idea is that the callback is called multiple times to send all the chunks, for one invocation of `streamingCall()`. Sorry if that wasn't clear.

BTW, Promise Pipelining is only involved in the client -> server streaming example.

> - in the same paragraph an "object-capability model" is introduced as a concept, but not explained

> - `sendChunk @0 (chunk :Data)` doesn't make sense to a beginner without explaination; what's `@0` and why do I care?

> - second paragraph, vocab: what's a "temporary RPC object"? Contrary to the precise, albeit unexplained vocabulary, in the first pragraph, this is vague.

This is a news post about a new release of an existing tool. You're expected to be familiar with the tool already, or if you are not, you can go read the rest of the web site to learn about it.

> - second paragraph: "think of this like providing a callback function in an object-oriented language", when it should be "in a functional programming language" (callbacks aren't OOP, they are per definition functional programming)

As others have pointed out, you are taking a very superficial and literalist definition of OOP and FP. That said, I should have said "callback object", because that's what the example actually illustrates.

> is this service on the server side, or the client side?

Cap'n Proto is a peer-to-peer protocol, not a client-server protocol. Either side can export interfaces and either side can initiate calls.

> - Why name a message, `Data` when it's clearly NOT a chunk of data in layer-3/layer-4, but rather a layer-7 artifact with retries and checksumming implemented to ensure complete and accurate message delivery?

`Data` is a basic data type in Cap'n Proto. It means an array of bytes. You can use any other type here if you want, it's just an example.

> > because the protocol can only achieve optimum throughput if a sender sends a sufficiently large quantity of data before being required to stop and wait until a confirming message is received from the receiver, acknowledging successful receipt of that data

> Which is not the case for Cap'n'Proto

What is "not the case"? The whole point of this streaming feature is to do exactly what's described in your Wikipedia quote.

> And there's no discussion of end-to-end problems like BufferBloat

BufferBloat is mentioned several times in the post (though I called it "queuing latency", describing the symptom rather than the cause).

> Going to https://capnproto.org/rpc.html immediately puts me off by inventing "time travel", calling it "promise pipelining" and showing an impossible trace diagram (you can't have messages go backwards in time).

"Time travel" and "infinitely faster" are obviously tongue-in-cheek claims.

> However, even when using the example of a file system (which is about as exception-intense as you can imagine), exceptions are ignored.

They are not ignored. Exceptions propagate to dependent calls. So if you send a chain of pipelined calls and the first call throws, all the later calls resolve by throwing the same exception. Eventually the caller waits for something and discovers the exception.

> it turns out they haven't actually performed the compile-time indirection, but actually block the calling thread

No, the calling thread does not block.

> like any random do-it-yourself-RPC framework out there.

Haha yeah that's me, just some amateur that knows nothing about network protocols...

> it turns out they haven't actually performed the compile-time indirection

Here's how much monadic control flow he actually has in Cap'n Proto: https://github.com/capnproto/capnproto/blob/77f20b4652e51b5a... — nothing except fields. Gut feeling was correct then.

https://news.ycombinator.com/item?id=22972728

No, that part of the protocol defines mobile code -- code which an RPC caller can ask the remote callee to execute directly on the remote machine. It's intentionally limited because Cap'n Proto is not trying to be a general-purpose code interpreter. Most RPC systems don't have this at all.

KJ Promises -- the underlying async framework that Cap'n Proto's C++ implementation is built on -- let you write arbitrary code using monadic control flow. But that arbitrary code executes on your own machine, not the remote machine.

It doesn't have to have a turing complete interpreter; it just have to be provably terminating, and you can build most use-cases as an active message.

What I'm after with the monadic control flow are the error-cases; let's say you have

> music.getPlaylist(ps => ps.userId == "u123").findTopSongs(10).enqueue(qInstance) => Result<C, Error>

it would be nice to see how this would be interpreted into an AST and executed as an active message on the server (receiver).

That said, I brought it up because the copy alluded to it. It's a great time sink to build an interpreter, even if it's only acting on a unit of work whose variant is strictly decreasing, just look at Linq-to-SQL and IQbservable<T> a decade ago.

> KJ Promises

Side note: another copy that greatly frustrates me, as I now try to find the docs on the above async stuff:

> Essentially, in our quest to avoid latency, we’ve resorted to using a singleton-ish design, and singletons are evil [linked to a page that crashes for HTTPS-everywhere users (me) with PR_END_OF_FILE_ERROR].

(Besides the broken link,) singletons are not always evil. I know you know this, I know the copy is tounge-in-cheek again and being ironic — BUT POE's LAW FOR CRYING OUT LOUD :D https://en.wikipedia.org/wiki/Poe%27s_law — "Poe's law is an adage of Internet culture stating that, without a clear indicator of the author's intent, it is impossible to create a parody of extreme views so obviously exaggerated that it cannot be mistaken by some readers for a sincere expression of the views being parodied"

...so KJ Promises; I can't find that mentioned. I only find Promise Pipelining; but that must be your Op:s that allow for field traversal? The site has this copy:

> [RPC Page] With pipelining, our 4-step example can be automatically reduced to a single round trip with no need to change our interface at all.

But I must be from another planet, because I really don't understand:

- first you have a pretty decent design of files that mimic local files

- now you instead showcase what rich messages look like, calling that a "[message?] singleton", linking to a broken site

- path string manipulation exists in every standard lib, it's not something we implement

- if someone wants to perform multiple ops on a file - let's say read a chunk of it * only Data needs to be resused for there to be no copies (contrary to the copy), but that's also a problem in the first decoupled example

  * often, almost always, when I read about what an RPC system can do, I'm not in the memory-management mind-set, so I don't care about re-allocating resources

  * caching is not a relevant solution, it's to the contrary; completely irrelevant in this context and it's detracting from understanding what you want me to understand

  * caches aren't error-prone when used right, like with immutable data, or read-through caches as transparent proxies can do, but all of this is beside the point

- then there's a discussion about "giving a Filesystem" to someone, when it's really all in my program

  * hard-coding a path is out of scope; that's about engineering process, not about the software. You ask "But what if they [have] hard-coded some path", I answer "yes, so what?"

  * what if we don't trust them (our own code?) — no, it's not an AuthZ decision locally, it's remotely, so you want to be explicit about the error cases here, but there's nothing about it — instead the copy says "now we have to implement [authN authZ systems]" — but again, this has nothing to do with merging small interfaces into a larger interface; it's a problem even with the small interfaces

- the section ends with the broken link and then the next section states "Promise Pipelining solves all of this!" — but noo, there's so much mentioned above, and the premise is unclear and I have no idea what exactly promise pipelining solves!

And then you go with the calculator example; but the file is large and I still don't know what "Promise Pipelining" means, where to look. I see a lot of construction of values going on and then a blocking wait (polling of the event-loop, but that's also not the point). There are so many bugs in that copy, that it's really hard to know where to start detangling it! With that copy, I would never in my life touch the underlying code! It should be fixed! (sorry, I'm getting into a state here, but that copy... wow)

And this is the kicker that seals the deal:

> Didn’t CORBA prove

Ehm, WTF? Why not contrast with gRPC? But also, why not clarify the above first so I can use that understanding myself? If I'm not a newbie at this, why do you mention CORBA? Do I look like the kind of person that would ask that question? It's demeaning to the reader.

And you mention object capabilities; that's a HUGE area of research, of which I've worked with no live system using it. But here it's casually mentioned, like building such as system is a walk in the park without explaining how.

---

So here I am after another frustrated 40 minutes on the site, and I still haven't found the docs on KJ Promises.

Very nice of people to down-vote because they don't agree with what I write; I'm not being mean or evil here, I'm just stating an opinion and trying to back that opinion up with clear reasoning, references and links to actual research. Down-voting then seems very much like group-think to me.

> It's not polling. The idea is that the callback is called multiple times to send all the chunks, for one invocation of `streamingCall()`. Sorry if that wasn't clear.

The point of my comment was exactly that there were many things in the release notes (and subsequent visits to the site) which were unclear.

> As others have pointed out, you are taking a very superficial and literalist definition of OOP and FP.

Precisely, which is my point; why use that terminology at all? It's neither specifically OOP nor FP, but if it's something, it's more FP than OOP to program with functions. Either way, it's super-superficial (pun intended! :)) and detracts from your message in your release notes!

> `Data` is a basic data type in Cap'n Proto. It means an array of bytes. You can use any other type here if you want, it's just an example.

But my point is whether the RPC frameworks garantuees complete messages or only complete "chunks" to arrive atomically?

> What is "not the case"? The whole point of this streaming feature is to do exactly what's described in your Wikipedia quote.

You're not solving the problem as far as I can read in your article, you've only pointed it out. In my second link, you'll see there are both a) second-, third- etc hop buffers to take into consideration at the sending side, b) there's timing to take into consideration, c) there's the whole "send a large message initially" heuristic, to take into consideration; all in all it turns into the end-to-end argument — you need knowledge on both sides to optimise this, and preferrably an out of band channel.

> "Time travel" and "infinitely faster" are obviously tongue-in-cheek claims.

It's obvious the literal claims are, but it may not be obvious to readers that your trace diagram is also tounge-in-cheek; at least clearly break with the jokey-attitude afterwards and perhaps show an actual trace diagram of how it works (because the trace diagram is only partially a lie; the pipelineing is still an interesting concept that I think people want to understand!).

> No, the calling thread does not block.

In your sample here you block https://github.com/capnproto/capnproto/blob/master/c++/sampl... to get the result. What I'm saying is that it's very unclear how you compose your flow without doing the blocking wait to get intemediate results in between RPC calls. (Exactly: you seem to lack a monadic control flow, specifically the tie-in that endofunctors would give you, and given this lack, you cannot handle exceptions out of band to the happy-path via Choice/Either/Result types)

> Haha yeah that's me, just some amateur that knows nothing about network protocols...

No, it's not nothing about you. I'm sure you're very competent. I'm sure Can'n'Proto is competent too; what I'm talking about is the way the framework is described and how the messaging looks to someone with close to two decades of programming experience. I hope you take is like this; as constructure criticism. I'm already very happy that you replied, because just as I believe you're a stanch proponent of your RPC framework and believe in it, I believe in my gut feeling when it comes to software — and my gut is telling me the above about your marketing message.

> But my point is whether the RPC frameworks garantuees complete messages or only complete "chunks" to arrive atomically?

Method calls arrive atomically. There is no concept of "chunks" in the RPC system itself; that was just what I used in the example. In practice by far the most common use case for streaming I've seen is byte streaming, e.g. large file downloads, so that's what the example used, but it's not limited to that.

> you need knowledge on both sides to optimise this, and preferrably an out of band channel.

Yes, Cap'n Proto has knowledge from both sides. The "Return" message from each call serves as an application-level acknowledgment that the message has been received and processed. This is enough information for the sender to maintain a send window that places an upper bound on buffer bloat. Calculating an ideal window size is tricky and the current solution of stealing the OS's choice is, as admitted in the post, a hack. But all the necessary information is there to do something better in the future.

> In your sample here you block https://github.com/capnproto/capnproto/blob/master/c++/sampl.... to get the result.

`.wait()` is a convenience method that means "run the event loop until this promise resolves". It can't be used recursively (so, can't be used from a callback called by the event loop), which means it can only be used at the top level of the program, e.g. in the program's main() function. Typically it is used in client code which really has one main task that it's trying to do. It turns out this pattern is a lot clearer and cleaner than the usual approach where you'd say "run the event loop now" and then have some other thing you call elsewhere to say "please break and exit from the event loop".

(Actually, with this release, the above isn't quite true anymore. .wait() can also be used in fibers. A fiber is an alternate call stack running in the same thread as the event loop, but not running the event loop itself. A fiber can call .wait() on a promise to switch back to the main stack and run the event loop until the promise resolves. This is a hack, not recommended for common use, but can be very useful for adapting synchronous libraries to work in an asynchronous program.)

Most KJ async code, though, does not use `.wait()`. It uses `promise.then(callback)`, which is exactly the monadic control flow you are looking for.

> someone with close to two decades of programming experience

Cool, I just hit the 30-year mark myself.

> I believe in my gut feeling when it comes to software — and my gut is telling me the above about your marketing message.

Exactly. Look, here's what I think happened here: You looked at this page, you saw the silly marketing, and your "gut feeling" told you that this guy is an amateur who needed to be put in his place. So then you started seeking out details that you could criticize as amateurish. But there's a lot to grok here, and instead of actually digging into every detail in depth, you started filling in the bits you didn't understand with your own assumptions. And in your assumptions, you assumed the details must be amateurish, because that's what your gut told you. So then you end up criticizing the amateurish details you yourself made up.

For example, you assumed that KJ promises don't use monadic control flow, when they most emphatically do. How did you get there? You looked at some code and found one example that didn't happen to use the monadic flow (unlike 99% of KJ async code), and then you assumed the rest because of course an amateur wouldn't know about monads.

This is why you were downvoted. (Not by me. Everyone else could see what was happening.)

To be fair, this is totally normal human behavior -- it's called "confirmation bias". This particular flavor of it is especially common among programmers and HN readers, in my experience. (I do it myself all the time.)

Now sure, maybe my post and the web site in general could have been better at explaining the details. But my advice for you is, next time, try to recognize when you don't actually know the details, and then try to assume the best possible details, or at least ask questions, rather than assuming the worst.

> Exactly. Look, here's what I think happened here: You looked at this page, you saw the silly marketing, and your "gut feeling" told you that this guy is an amateur who needed to be put in his place. So then you started seeking out details that you could criticize as amateurish.

Again, I'm not attacking you as a person or your work, I'm attacking the shoddy copy. But yes, this whole thread is about what the first impression of your web site is like, and it's amateurish. Perhaps you're someone who can live with that, and how that results in lost conversions (?) from visits, maybe you're not. In any case, from running businesses myself, I know that getting clear and honest feedback is worth its weight in gold.

> instead of actually digging into every detail in depth

Which is precisely my point that I won't do. If you can't communicate clearly to your visitors on their first visit, but expect them to dig into the details, you've lost that battle.

> For example, you assumed that KJ promises don't use monadic control flow, when they most emphatically do. How did you get there? You looked at some code and found one example that didn't happen to use the monadic flow.

Again, what I found has nothing to do with you. I'm yet to actually have seen the examples you mention in your docs, and the pages I've linked to and the calculator sample doesn't show your statements to be true (only based on that). Perhaps, instead of being so tounge-in-cheek on your introduction page, you assume the visitor is not an amateur, and show what you got (you can move the funny bits to after the introduction, when the reader is in on the joke)?

> The "Return" message from each call serves as an application-level acknowledgment that the message has been received and processed. This is enough information for the sender to maintain a send window that places an upper bound on buffer bloat.

THIS; is what kind of copy you could use. Also not using a confusing `Data` record name, which conflates L4 and L7 semantics (even if that data is identical to memory of a record), not using `my-` prefixed variables, but building a use-case like "suppose this was for streaming music", or something.

If uptake is an aim, focusing much more on other languages than C++ is probably wise, too. And some diagrams to explain what trade-offs have been made (can you load-balance CapNProto like you can gRPC? Is -Proto, Protobuf, so I can use an existing schema registry in my "enterprise"? Is it safe, "like HTTP2" — the newbie stuff in relation to the world around you).

> Cool, I just hit the 30-year mark myself.

Time flies, eh? ;)

> But my advice for you is, next time, try to recognize when you don't actually know the details, and then try to assume the best possible details, or at least ask questions, rather than assuming the worst.

I'm aware I don't know the details! I've said so repeatedly; if I read the details while I have a conversation about how someone who's never encountered Cap'n Proto before, reacts to its material, I'm not that person anymore.

From the above person's perspective, you have a bit of a problem with marketing; I've actually bounced off your front page about three times the last six (maybe?) years and the copy/messaging has always put me off. I'm sure I'm not the only one. Just ping me if you actually do like having someone review that copy for you.

> lost conversions

I'm not selling anything. This isn't a business.

> I've actually bounced off your front page about three times the last six (maybe?) years and the copy/messaging has always put me off.

Working as intended. :)

Hasura on Postgres works really well too.

Does anybody know why the USA hogs the toplevel domain? It's not the only government in the world. It would seem more just to make it more like .com than .edu.

Obviously, because of history of Internet deriving from Arpanet. The whole domain name structure grew out of the needs of the US government, even if the .com domain was largest TLD from the start.

Nope, .gov belongs to the US, so they get to hog it.

It's a historical vestige, the Internet started out as a U.S. government-sponsored research network, so they built it for their own needs. There's absolutely no reason to them to give that up.

It's a perk from building the internet. Early bird gets the worm etc

In addition to all the siblings, government isn't always spelled with 'gov' so it would be useful to the subset of countries where those letters make sense. Compared to say Mexico with http://gob.mx .

Because they came up with it.

Like their free DNS service that supports DNSSEC, Rotor (CDN JS optimisation), Let's Encrypt integration out of the box, low latencies where the users live, etc

If you really care about the low latencies, you might want to not pull up megabytes of JavaScript libraries to simply render up a few bytes of text.

I have a single server on another continent, yet my pages load several times faster than any of these Cloudflare-powered websites served from another part of town.

Let's Encrypt itself is an artificial construct that your homepage would hardly benefit from, and where Cloudflare reaps the most benefits from by having it be a selling point that noone outside of commercial entities ought to need in the first place.