Matt: 0:00 What's up, wizards? This is a joyous occasion. We have Colin to talk to here. Colin is the creator of an amazing library called Zod. He's a TypeScript wizard extraordinaire.

0:12 I'm really excited to talk to you today, especially about generics, about conditional types, about using TypeScript in application code and library code, and getting an insight into what it was like to build Zod and the design decisions that went into it. Why don't you introduce yourself? Also, you help with tRPC, is that right?

Colin: 0:35 Yeah, that's right. Thanks, Matt. Thanks for having me for this. I'm actually very excited for this conversation, and just excited to see what you're doing with Total TypeScript. It's so good. Just immaculate stuff. I made Zod, published the first version March 2020. It was underway but then was very much like a pandemic kind of project as it turned out.

1:03 I graduated college in 2016, I did a few years of indie hacking stuff, so I was building apps with Node React, basically starting as soon as I graduated, and held out on getting a real job for a while. Whether or not that was a good plan or not, whether I'd recommend it to people is a different question.

1:27 The short answer is probably not, but it was fun in terms of just getting exposure to actually building real things and getting into the JavaScript ecosystem early-ish. That was all pre-TypeScript for me. I only wrote my first line of TypeScript in 2019, which is when I was doing startup stuff, like I mentioned, and was lucky enough to get into YC, was a solo founder.

1:59 This was the Winter '19 batch, so four years ago now. I was like, "I'm going into this as a technical founder, it's just me, I need to make every technical decision." Have zero tech debt, make all the right calls out of the gate, and just build this to be totally future-proof.

2:22 Needless to say, I was like, "It's finally time to learn TypeScript. I need to do this. This is the new hotness, and it just adds this," I'm getting ahead of myself a little bit, but TypeScript just makes your code so much more maintainable.

2:38 Not an original idea, but I loved the idea of having my code base just feel crystalline, where you change a type signature, whatever, it tells you exactly all the places that you need to change your code to actually now be in conformance with that interface, whatever it is.

2:58 It's beautiful to me. Those red squigglies that are on the total TypeScript page, the huge, flaming red squigglies are really your best friend. I'm not sure I fully appreciated that at the time, because it did feel like you're working against TypeScript for a long period of time.

3:21 I didn't have the knowledge at that point of generics or things to express all the types that I needed to build this app. I ended up doing a lot of as anys, taking a lot of shortcuts, and loosening up my type safety throughout my application, because I didn't have the ability to represent all these things.

3:50 Specifically, what I was doing, which is possibly...It's funny, I said, I went into this trying to make all the right technical decisions. I made an astonishingly terrible one right out of the gate, which is that I was trying to be clever. I was like, "These relational databases with their migrations and all this nonsense, that's going to slow down my development speed way too much. This is not acceptable."

4:14 Being very clever boy that I am, I decided, "Oh, I'm going to go for a graph database. I'm going to build this mission critical, medical software fully backed by a schema list graph database." I was using Neo4j. I like graph databases, they're nifty for a lot of use cases, but they don't let you...

Matt: 4:39 They didn't live up to the crystalline promise that you went for, yeah.

Colin: 4:42 The least crystalline possible choice. Yeah, exactly.

Matt: 4:47 Was it this idea that you wanted a perfect code base, a code base that's lived up to the expectation of being maintainable? Was that what led you to build Zod?

Colin: 5:00 Yeah. I think that's definitely true, because Neo4j is not a relational database. It's a schema lists, and it doesn't enforce any types for you. You can't even say the name of a person object should be a string. Well, you actually can now, but you're pretty limited in the kinds of constraints that Neo4j will enforce for you.

5:24 What I ended up doing is every single bit of data that I wrote into my database, I needed a way to pass it through some validation checks to make sure that it was in conformance with what I wanted the types to be. The way that I needed to represent that was with object schemas. These object schemas also had to be recursive because I built my own Neo4j ORM. That was terrible, by the way.

5:58 You could just pass in arbitrarily nested objects, and it would convert that into crazy, hairy Neo4j Cypher queries -- the Cypher's the language for Neo4j -- and write all that into the database in one go. You needed to do pre-validation for that to actually work. I looked into every validation library that existed. There was a lot of good ones at the time.

6:32 Whether or not I should have built Zod instead of trying to contribute to another one is something that I've thought about a fair bit. Ultimately, I think it was probably the right call. The big two at the time were Yup, which was saddled with a bit of a bloated API because it was specifically trying to be a slicker implementation of Joi.

7:02 They already had their API spec out for them, including a bunch of stuff that I got rid of with Zod and no one has ever missed, seemingly. Then the other one was IOTS. We're going to talk a bit more about Zod's chainable interface here, but IOTS despises that because it is so steeped in the functional programming paradigm.

7:23 It's all about function composition, you end up having a bunch of nested function calls instead of a bunch of chained method calls, basically. I wasn't going to be able to strip out half of Yup's API in a PR, and I wasn't going to be able to convince Giulio Canti to add a bunch of methods to his gorgeous, beautiful functional programming library.

7:49 I think the goals for Zod were probably fairly incompatible with some of the big libraries out there now, or that existed at the time. My recommendation would usually be, everyone wants to make their own library. It's really hard and just time-consuming, and you don't necessarily get...I'm super happy that I did it, and I get a lot of value out of the reaction to Zod and how many people like it.

8:26 I know this is not what the course is about, but in terms of any reasonable, tangible value back to your life, I think, not an original idea, but most open-source maintainers would probably say, "Oh, yeah. Well, it's been a lot of work and..." I'd say, minimal payback or payout. I don't know.

Matt: 8:45 Interesting. There's a tension there because when you build something as big and complex as Zod, then it needs to be applicable to many, many different use cases. What it sounds like to me is the birth of Zod came from, you had a need in your application code that no library out there was properly going to meet.

9:05 Nothing quite mapped onto what you needed. Have you found cases where that's been true on a smaller scale, where you've built, let's say, a set of functions or a wrapper around something? That's a lot of what Total TypeScript is, is giving you the knowledge to be able to be confident enough to build those little abstractions that you need.

Colin: 9:28 Great call, yeah. All of the things that I learned over the course of building Zod, I use very frequently now over the course of normal application writing, for sure.

Matt: 9:39 Interesting.

Colin: 9:40 It's indubitable that having at least a basic grasp of generics and conditionals is just necessary now if you want to be able to write code that is, say, fully dry. That's a big part of it. You can do whatever you want in TypeScript without generics if you're willing to have a lot of duplicative stuff or to lose a bit of type safety, things like that.

10:07 If you wanted to build a function that, let's say, you're using React Query. React Query gives you back this pretty hairy generic object when you run useQuery. If you want to be able to take the result of any call to useQuery, that is what the hook is called, for React Query. It's been a lot...

Matt: 10:29 Yeah. Something like that. Has been a lot for me as well.

Colin: 10:31 Yeah, I know. If you want to be able to have some function you can use in your components that they perform some modification to any results of that comes out of useQuery, you need to be able to express basically, that needs to be a generic function, where you're basically inferring the generics that are attached to that React Query result.

11:03 Then you can do a modification, probably use a generic type at the output of that function to describe the change that you made, and then that way you don't lose any of the type safety. Exactly.

Matt: 11:13 It just chains, and chains, and chains. You're constrained by the external library, by what sort of generics you should be applying. Was that something that you thought about when you were building your generics?

Colin: 11:28 Definitely not. Mostly because I more or less learned TypeScript over the course of building Zod. Everything here was new to me and I just spent a lot of time in the code base of Yelp and IOTS specifically to try to figure out what approaches they used to implement these things, because it was not remotely obvious to me any of this TypeScript stuff when I set out.

11:54 I mentioned I was fully playing JavaScript up until I started building this application with Neo4j like an idiot. It was a learning process along the whole way. I did not know a lot of Typescripts going in. One of the early issues on Zod, this is a testament to how newbie I was.

12:16 It's slightly embarrassing, but I think it's a good story just because the people who build libraries like Zod are not any better than you, is the real moral here. One of the first issues on Zod, which I published version one, or version whatever -- I guess it was version one -- wrote up a blog post and posted it to Hacker News.

12:40 Someone saw it on there, was clicking around the code, opened an issue of saying, "Yeah. I'm trying install this from MPM. It looks like you just shipped the raw TypeScript files in the MPM module instead of transfiling them." I was like, "What's transfile?"

12:56 I just had shipped the TypeScript files because I didn't know any better. The fun part now is, if you're using, not to shill for my employer or other run times, but Bun and Dino now can just run raw TypeScript files, which means that we're moving towards a world where you can actually do that and it'll work which I'm very excited for.

Matt: 13:16 That's good because you'll never have that issue again, right?

Colin: 13:19 Exactly, yeah. Never transfile anything. That's the goal.

Matt: 13:24 That's amazing because I think a lot of how people think of advanced TypeScript as some sort of supernatural skill, something that even some people would think that you're born with the level of cleverness that you need to understand it often, from talking to you, from talking to Alex, from TLPC, it's just hacking around until you find something that works.

13:46 In the spirit of hacking around, let's actually go into Zod's code base and let's take a look at, I really want to dive deeper into the decisions that you made and start talking specifically about TypeScript syntax, start tying this into the stuff that folks might have learned in total TypeScript.

14:06 I'm inside Zod's code base here, which I'm sure you are tremendously familiar with. It's just a single package, right? Just a single package .JSON. There's not much of a monorepo here. It looks like everything is basically inside source and type.ts. I don't have line numbers on, but this is a big old file. This is a few thousand lines long.

Colin: 14:28 Yeah, it is.

Matt: 14:29 Basically, everything's in here.

Colin: 14:32 Yeah. It's all in there. For a very practical reason which is in the first year or so of Zod, we ran into crazy errors with cyclical imports when Zod was a multi-file thing. Depending on whether you used common JS or ESM, you have slightly different module resolution patterns there.

14:56 Back when we had one file for each Zod subclass, we basically had a very hard time getting it to work across all module systems until we put it all in one file because TypeScript is smart enough with a bunch of classes that reference each other, to just, figure all that out, but the module systems aren't necessary.

Matt: 15:17 That makes sense. Let's start with, because I think one thing that people, when they're thinking about building these libraries will think, should I use functions or should I use classes?

15:29 You've chosen classes with Zod, and pretty much everything in Zod there's a lot of things which are not classes, a lot of things are basically just methods inside these classes. The main types you've chosen to represent with classes. Why was that?

Colin: 15:43 The super brief answer is, when I was doing this, I was not aware of any other way to really represent that. Now, this is what IOTS does as well, and we can talk the fact that you use classes here lets you represent complex types in a pretty concise way.

16:07 This is the base type for all Zod classes, as you said. It looks pretty weird here. You can see there's three generics. Back when Zod started, there was only two, which was output, I think it was called type at the time. The name doesn't actually matter.

16:24 That first one is the inferred type of your schema, and that is the type you get back out when you call parse on something. Then Def is just a random name I made up for essentially the internals of a given schema.

16:40 Something that's interesting about Zod that I think it was a good design decision, but it does make implementing certain things pretty tricky is that, yes, Zod infers the TypeScript type that your schema is representing, but also the entire schema hierarchy, things like ZodObject, obviously, they are compositional schema types. They contain additional schema within them.

17:09 That entire type schema hierarchy is fully typed all the way down. You can introspect the shape of a ZodObject, for instance, and you get back the exact same type signature of whatever you parsed in here.

17:23 If you did Z.object, name Z.string, something like that, then you know you could just take that example schema and then you do example.shape.name. That just gives you back that same Z. string that you parsed in originally.

Matt: 17:42 Wow. That's strongly typed there. That's really nice.

Colin: 17:46 Exactly. Strongly typed. There's a universe where Zod doesn't have this property where every schema doesn't know, it's like a black box. You can't introspect it or see its internals. All it does is track that first generic, which is the TypeScript type that it represents. It would make some things a lot easier.

Matt: 18:09 Got you. To pause you there and just talk about that a little bit more, because that's a massive thing I talk about in the course as well, which is choosing the right shape for your generics. Choosing what information to store in the generic slots, basically. It looks like here you have this ZodRawShape. Is that right?

Colin: 18:32 Yep.

Matt: 18:34 This is for the create method, which is when you create an object. If you could explain more about this.

Colin: 18:43 This is a interesting pattern. That second generic that we're looking at, the Def, it's different for every subclass. All the Defs for every Zod subclass conform to something that I think is just called ZodTypeDef, which is really simple. I think it only has one property on it.

Colin: 19:05 This thing. Yeah, I see it.

Colin: 19:08 You can find that. There we go. Every schema has a Def, and that Def contains these two properties. Of course, the subclasses that extends ZodType, they have additional properties on top of that. For instance, this ZodObject contains that shape property in the Def.

Matt: 19:24 Got you.

Colin: 19:29 It is a weird pattern where there's the constructor for each subclass. Actually, there's only one constructor. It's defined on the base class. It just accepts the Def, and then it just sets this .Def to whatever you pass in.

Matt: 19:45 Got you.

Colin: 19:47 You can see that. If you search constructor, you shouldn't be able to find it. I think it's, for some reason, just halfway down the definition of subtype for unexplainable reasons.

Matt: 19:55 For legacy reasons, yeah?

Colin: 19:58 Right, yeah. The only thing that matters here, I am binding all the methods because there's ways to have weird, unexpected behavior if you don't do that, but essentially, all it's doing is assigning to this .Def, and I don't override that constructor in the subclasses. That's all that this needs.

20:19 What I do do is I have those static methods that you referred to. They're all called create, and they're static. That's basically a function where, instead of having to pass in the full Def object for all these things, it is a nice clean function that only accepts the bare minimum required for this given subclass. I think, we're inside of, here is ZodNull. It would be weird whenever you...

Matt: 20:48 I see.

Colin: 20:49 ZodNull doesn't need really anything. You're only ever called z.. You could pass in some additional parameters or a config into that, a custom error messages, things like that. That's what this function is doing. It's like these raw create params are things that can be defined on any schema, but then it calls the ZodNull constructor here.

21:16 These static methods that are declared on all these classes, those are the z.whatever functions that people are used to. You can see it right here. I will say, export ZodNull.create as nullType, I guess, is what it's...I have to do the as here, because you can't export something called . You can't name a variable .

21:45 What you do, I just have nullType and then I do the export as. That lets you export something z.. That's a...

Matt: 21:53 We get a sense for then how the things are constructed. It seems like you're using classes here in a object-oriented style as well. You have a base class and then you have all of these things which inherit from the base class. It seems then that you can do some cool stuff here because you can also extend the generic stuff as well. Is there a good example for maybe ZodString or something?

Colin: 22:22 ZodString is perfect.

Matt: 22:23 Yeah. Where is that? That's somewhere up here.

Colin: 22:27 If you search class ZodString, then you can find it easier.

Matt: 22:30 Here it is. This is basically saying, ZodString extends ZodType string, ZodStringDef. Here then, we're using ZodType. We're saying that the output is a string, and this StringDef looks like this, which is the Def that you're talking about.

Colin: 22:50 Exactly. You can see that it is extending that base interface, ZodTypeDef, there and to add the properties that are specific to ZodStringDef. In this case, has a big array of checks, which gets appended to as you call .startsWith(), .min(), .max(), whatever methods you want to call.

23:11 Coerce is a brand new one as of what, two, three weeks ago that is defined on all of the primitive types. That will do some pre-parsing coercion on everything you pass in. Something that's interesting here about the subclass approach is that it gives you these complexity boundaries, which is really nice.

23:33 When you create a ZodString schema and you hover over it in VS Code, it just says ZodString, which is really valuable, I think. Hover over z.ZodString. That's really nice. This subclassing is one of the few patterns in TypeScript that lets you represent complex things, but then it's hidden in the IntelliSense, basically.

24:09 If I represented this instead of with subclasses, like with the type keyword, let's say I have a type, which is all of the methods to find on the base class, then I have additional methods or something that are specific to the string schema, and then the way that I represent the result of z.string is I union those together, that is similar to subclassing in some sense.

24:40 You've got the base methods from one type, you've got all the other stuff defined in another type, and you mush them together. That would be equivalent, but the way that TypeScript would represent that to you is this horrifying union of two very complicated object types, and it would just be completely unintelligible. Then, that's what you're looking at.

Matt: 25:01 This lets you be really clean with what's actually output in the IntelliSense?

Colin: 25:07 Yes, exactly. There's this two-level generic pattern where ZodString is simplified here, because ZodString itself is not generic, which is interesting. It's just ZodString, no generic parameters after that, but it extends that superclass. Then you pass in all of those generics that correspond to in this scenario of what the Def type is and then what the actual inferred type is, which of course is string.

25:40 Currently, ZodType has three generics, output, Def and input. That's a pretty weird ordering when you look at it that way, where the two inferred types are split in first and third. That's because at the beginning, Zod didn't have any concept of tracking the input type, because transformers didn't exist or .transform.

26:04 It was tacked on as the third generic because ultimately, the ordering doesn't matter and I just wanted to minimize breaking changes to the generic ordering here. Now, every Zod schema actually tracks two types, which is the output of the thing and the input of the thing. If you do a .transform, then those two diverge. If you don't, then they're always going to be the same.

Matt: 26:27 I wanted to raise something here, which is this reminds me of a lot of work we did on XState, because on XState, you had something like createMachine, like this. I'm not sure if you've ever used XState. It's like a state chart, state machine library.

26:44 It had two generics in here, TContext, TEvent. We went through a similar process as you, deciding we probably need to have a third one on here, except that we were letting people call it with arbitrary things here.

26:58 We were letting them pass in different events and whatever here, that sort of thing. This was valid to us, because in Zod, you don't need to do this. You don't need to pass type arguments. I imagine it's in fact just totally not recommended.

Colin: 27:16 Yeah, definitely not. The only maybe two scenarios where you ever need to pass in type hints like that would be for recursive types, unfortunately, which is something that is a thorn in my side. The fact that you can't declare a recursive type without any type annotation very easily. I have some ideas around this. This is going to be either like a Zod 4 or a new library.

27:47 That's a little tease for the future maybe, about how to be able to actually declare recursive types in a clean way without having to separately declare a recursive interface and then cast your schema to ZodType and then manually passing that in.

28:07 The only other scenario where you would need to use the type in is there's something in Zod that's an escape hatch, which is z.custom, which is you basically call z.custom, the function, and then, in the brackets, you can pass in the type that you want this schema to represent. Then optionally, the run time argument there.

Matt: 28:27 The value.

Colin: 28:28 The actual parameter you pass into the function would be a validation function that accepts an input. You can just run whatever checks you want.

Matt: 28:35 I see, which accepts input. Got it. Then you accept this because there's no real way to define it otherwise. I guess you could return it there maybe, and say...Does this return the type, or does it just throw an error if it's incorrect?

Colin: 28:58 That's a interesting question. That function that you pass in, I'm pretty sure it's just supposed to return a Boolean, which is whether or not it's valid or not. I don't think there's any real way for it to know whether you've done an appropriate run time check for the type that you passed in, if that makes sense.

29:17 Now that you say it, that should be, except a function that is, I can't remember the name of what these are called, but the type assertion functions, like X is Fish or whatever.

Matt: 29:30 Type predicate, yeah.

Colin: 29:32 Type predicate, thank you. It's a good thing that you're the one making this course and not me.

Matt: 29:37 I had my head in it for too long. This is something that tRPC accepts, which is you can say it returns the thing that it's validating.

29:48 You would say, if type of inputs equal equals objects -- Oh, Christ, can't type -- and input.something, whatever, whatever, and then you actually return the thing as whatever. Then it infers the type from there, but as an alternative, you've chosen this thing, and then you return a Boolean, I guess.

Colin: 30:17 Exactly. I think I like Zod's maybe just slightly better just because you don't have to write that type name twice, I suppose. Both of these, both passing into the brackets or doing the as are both just typecasting.

30:38 There's not necessarily a difference in my mind too in terms of actual type safety here. This is ideally the only time that Zod has an API that actually expects you to pass something in as a type hint there because the whole idea is inference.

Matt: 30:56 That's nice because it means you can be free to just flexibly change your generics, which is pretty cool.

Colin: 31:02 I was being extra careful in tacking on input as the third one there because now there are third-party libraries that are built on top of Zod. Additionally, Zod has a very open policy where, if you look in types.ts, every single thing is exported, including a bunch of stuff that others would consider internal.

31:25 I don't know to what extent this is much of a contributor to Zod's success, but that full openness and giving people access to everything they need to build third-party stuff on top of it, of arbitrary complexity, is something that I care deeply about. The unfortunate ramification is that, for some of these complex things that allow you to convert a Zod schema to a JSON schema or something, those rely...

Matt: 31:55 All your internals are exposed.

Colin: 31:56 Yeah, exactly. It's a source of contention, where I considered the public API of Zod to just be the z.whatever functions, the methods, and then maybe things like the Zod era map type that are intended to be consumed. Anything else, there are technically breaking changes in every minor version of Zod.

32:21 Maintaining true SemVer if I tried to do that, the way that tRPC did back in the earlier versions, it would be on Zod version 10,000 I think right now.

Matt: 32:36 That's interesting, because obviously if you add a fourth generic onto this, I can see you've done some defaults here, imagine if we add a T4 or something. Look at the red line in my map. There's red lines everywhere. That refactor in this code base to add an extra generic, I remember it being extremely painful in X state.

Colin: 33:06 It would be a big one. A lot of times, you could see for ZodString, I didn't bother adding in the third generic because I knew that it would default to whatever I had for the first generic, which you see there with the input equals output.

33:24 I just rely on those default values just for the sake of code brevity. You could put a comma string after that right there and it would not change anything. That's exactly equivalent. I just didn't do it because of sheer laziness, because I knew that that default value is going to get set automatically.

Matt: 33:42 That's a nice use case for default values, is that you can just save a little bit of code and it adds a little bit of logic because this one is relying on the results of output, which is cool.

Colin: 33:54 Exactly. You can get fancy too where your default values, you could use a conditional type in there that basically return a modified version of earlier values.

Matt: 34:06 Let's show that trick because that's really cool. If we say T1 extends string, and then we say T2 equals T1 extends "Hello," "Goodbye," "Hello." That doesn't actually matter what we return here. Actually, no, we can return T2. Then we can say, in fact, I'll just say Beatles song. It's how all my tips get created, by the way. It's just all of this stupid improvisation.

34:43 Type results equals Beatles song, and then we can actually pass in "Goodbye." Then we get "Hello" back. If we hover over that...Oh, that's interesting. It doesn't show the thing that's actually locked in here, but all of the logic is actually being done inside here. This is declaring a local variable in the generic.

Colin: 35:09 Exactly, yeah. That is interesting. When you hover over result, you see the full definition of Beatles song? There we go.

Matt: 35:18 Only when you hover over Beatles song do you get the full whack.

Colin: 35:22 Got you. Makes sense. That's a well-done made-up-on-the-fly example there. If you change that to hello, then you get goodbye back. It's very nice.

Matt: 35:32 The assumptions, there it is.

Colin: 35:36 In all of my examples, I just start naming everything tuna for some reason, so this is much more classy.

Matt: 35:45 Interesting. You always make your examples around lunchtime, I imagine so.

Colin: 35:48 I think so. Something like that. I haven't had tuna in a long time though, so I don't know why that stuck in there. Actually, I do.

Matt: 35:54 That's interesting.

Colin: 35:57 Learning programming, I watched a lot of "thenewboston" tutorials, and the guy who ran that channel loved tuna, or all fish-themed variables, frankly. It's a good enough gimmick.

Matt: 36:12 Another thing I want to touch on, which is you mentioned casting earlier. It is too late to use it as a segue, but I'm going to use it as a segue anyway. How many a's and e's do you think there are in the code base?

Colin: 36:28 64? 62.

Matt: 36:27 64, 62. Do you see this as a problem or what use do you see for as any throughout the code base?

Colin: 36:40 I don't see it as a problem, not remotely. This is funny. When I've contributed to some other libraries, a lot of times my contribution will involve any somewhere along the line. People will send me back the little snippet of the TypeScript handbook. That's like, "Try to use unknown instead of any," which is adorable.

37:04 It's a good advice for the end user, but for library authors, you use as any when you're smarter than TypeScript is. Smarter than TypeScript is at inference, I suppose. That happens a lot, as you can see here, where anytime you're writing a function where you take an input, you have a conditional type at the output, where you transform the Input into the output somehow.

37:33 Usually, TypeScript won't be able to look at your run time code and figure out what that transformation is that you're doing. Especially if it's a sequence of imperative steps that use arcane bits of JavaScript functionality. It's not going to be able to infer that properly, even pretty basic things, like .map, if you're using that.

37:58 TypeScript is notoriously bad at figuring out what the result of array.map is going to be. Any function where I actually have a type there for the result of the function, here, it says, "ZodOptional this," if I'm doing that, that's me basically saying I've told TypeScript what I want the result of this to be.

38:25 I no longer care at all about writing my code, such that TypeScript is able to infer what the result of that function is. I'm just going to write my run time code in this method and call as any at the end. Is that actually working? Well, that's a...

Matt: 38:41 Oh, my gosh.

Colin: 38:42 That's astonishing.

Matt: 38:42 That's fascinating.

Colin: 38:45 Some of these...

Matt: 38:45 Oh my God, I'm going to make a PR right now.

Colin: 38:49 I'll send you that down to 60.

Matt: 38:50 the docs as well. That's interesting. Like this, this is exactly what you're talking about, which is you're saying this is the return type of this method no matter what this says, no matter what this type is.

39:05 Another way you could do it would be to say ZodOptional this, although, hello, that's not working there. Blah, blah, blah, blah, blah, and plus here is return type. Oh, yeah, you've probably got some different settings.

Colin: 39:16 Yeah, so that is a thing where you can't infer it, because ultimately this is a recursive type here. The general rule of TypeScript is you can't just with code without any typecasting, you're not able to basically represent recursive structures in a way that can be inferred by TypeScript. This is a real nitty-gritty kind of thing.

39:42 You need to have a true return type on this method, because this is a method on the base class that returns an instance of a subclass of the base class, which is, we're talking about the cyclical import errors here, this is why that base class has methods that return subclasses of itself.

40:03 The base class was importing from ZodOptional, here's Zod, the ZodType file would be importing from ZodOptional to return just for use in this method, but of course ZodOptional has to import the base class because that's the class that it extends. You had...

Matt: 40:19 This is...Sorry, carry on.

Colin: 40:23 Basically, you just had a huge, massive cyclical imports, and I literally couldn't solve it other than putting all of the types in one file. Just wild.

Matt: 40:34 What I'm really looking at here is this or this here, because this is on the type level, what you're talking about, which is, you're basically saying whatever this is, whether it's a subclass or not, then make this optional. That's really, really cool. I don't think I've actually seen this pattern before. Was this something that you knew was possible when you started doing like the classes approach?

Colin: 40:58 No. This is the kind of thing where I had to...I mentioned, apparently, this episode is just going to be me telling people about IOTS, but this is not a pattern that IOTS used, because they don't do methods. They didn't have a need for this. This is something that I had to figure out through trial and error.

41:21 Because this is a very messy recursive thing here, there were periods of time where Zod would just encounter what is one of the most terrifying errors you can ever encounter, which is type is excessively deep errors, which is what happens when you accidentally define basically an infinite loop in the type system.

41:44 TypeScript tries to resolve it down to a depth of 500 iterations. If it can't, then it basically bottoms out and just sets the type to any, and then throws an error a lot of times, which is this excessively deep error.

42:00 Going back to the any versus unknown thing, any was actually necessary to break some of these infinite type resolution loops, because the difference between any and unknown is that if you have a variable of type unknown, you can't assign it to anything. It's not assignable no matter what.

42:24 You can't pass it into any function, no matter what that function expects. Whereas something with type any, it's assignable to any type. You could pass it into any function basically, regardless of what the parameter types that the function expects are.

Matt: 42:40 Got you. You can use that to break the loop basically.

Colin: 42:44 Yeah, exactly. It just like, when TypeScript is trying to figure out ZodOptional this, that means we're taking the subclass ZodOptional, the one generic parameter that ZodOptional expects is just another Zod schema.

43:01 It's this weird thing where now ZodOptional has all of its methods that are attached to it, but then there's a variable on ZodOptional that inside of the Def that will give you back the original schema that was originally made optional. That's like this recursive type here.

43:23 Honestly, the reasons, like the times where this successfully deep error come up are still mysterious to me a lot of times. You just need to like mess around with it until TypeScript ends up happy. Can you search for the phrase extends this anywhere in here? I'm really hoping this isn't here. That's great.

43:46 For a while, there were certain issues with recursive types where instead of just passing in this, like ZodOptional this, I would have to do a thing where I would have an additional generic parameter that was T extends this, and then I would have to pass in T, which somehow it would be on optional here exactly right there, which also works, I believe.

44:12 It's just like this egregious hack that somehow broke these loops in TypeScript. Ultimately, I think a few versions of TypeScript ago, it got a little smarter about resolving these sorts of infinite loop things.

44:27 Just lots of what Zod did back in the day was very not kosher, recursive, all these hacks to do recursive type modifications that TypeScript did not officially support, but there were these hacky workarounds that you were able to do.

44:46 Things like the partial or deep partial methods on ZodObject, they're still pretty hairy and pretty incomplete in terms of their implementation, because they rely on these crazy recursive conditionals that I wouldn't recommend.

Matt: 45:06 Got it. We've been deep in the weeds in terms of library concerns, in terms of what you need to think about to get around the TypeScript compiler. We've got a bit more of an understanding now about Zod structure, why you chose classes, why you chose the generic shapes as well that you picked.

45:28 I wanted to ask you about one particular pattern, which is a little bit more high level, which is I can see that there's a type in here called ZodString check, which is just a pure discriminated union, basically min, max, length, URL, UID.

45:45 This represents the type of checks that you can put into a ZodString. You add, you call min on it and it adds this kind min thing. This looks like it's only built for internal use. I was wondering why you chose a discriminated union to represent this, and what you think about discriminated unions in general and their use cases.

Colin: 46:10 I love discriminated unions. They're one of the best things about TypeScript's type system. This goes back to the crystalline code base concern as it applies to Zod, which is that once you have a discriminated union, TypeScript gives you the tools you need to make sure that you handle every possible case.

46:35 If you don't handle all those cases, then TypeScript will give you that red squiggly. If you go into the ZodStringParse function, you'll be able to, or rather _parse for ZodString...This'll be a little tricky.

Matt: 46:53 Baby, here we go, here we go.

Colin: 46:54 I think it's up.

Matt: 46:59 You keep talking, I'll find it. Here we go.

Colin: 47:01 Cool. Then you can see that inside of here, we do the basic checks at the beginning. We just make sure that the type of input equals string. Then further down is where we actually loop through all of the checks, which every check in this array matches against one of those types of the discriminated union.

47:25 There's an exhaustiveness check on here, where I make sure that the logic is implemented for every kind of thing here, and then I use this util.assertNever. All it does is that it accepts one input, and that input has to be a type never.

47:42 TypeScript is smart enough where if you have a big if-else statement that covers every single possible value for the key in a discriminated union, then what's left in that final out statement is of type never. We use that here.

47:57 If there was any remaining element of that discriminated union that we hadn't handled, which you would comment that out or something, then you'll start seeing the...yep, there it is. I love exhaustiveness checks. They just make everything feel great. Practically speaking, make it harder to introduce bugs, and make it easier for people to submit PRs.

48:21 The checks like this, in particular, a lot of these different types of checks, have been submitted via PR as opposed to implemented by me just because people know what methods they want on their string schemas. That's been a fertile place in terms of contributions to Zod, which I love.

Matt: 48:40 This makes it easy to extend, it makes it type-safe to extend. This stuff seems like it would belong in application code as well.

Colin: 48:50 Absolutely, yeah. Could not recommend discriminated unions enough there, especially since when you have...You could represent this in a pretty messy way with just a bunch of optional fields or something, but then you end up having to do all sorts of things.

49:13 It's much easier to be able to just do a simple check, and then inside of these if-else statements TypeScript gives you the exact type of that sub-element of the discriminated union and it's just so clean.

49:25 Whereas you can see there's a lot of shared fields here. I guess you could do something where instead of kind being a string literal, it could just be "kind:" string, and then value, and inclusive, and message could all just be optional fields.

49:40 You actually end up with something that more or less works, but then what you'll have to end up doing is not all of these subtypes here have, for instance, the value field on them.

49:54 When you don't want the value property there -- it's going to be there anyway -- and when you do, it's going to be optional, which means that you're going to have to use an exclamation point to basically make it required so that you can use it without it being a bother, and so TypeScript is happy.

50:10 Discriminated unions, massively useful across a ton of use cases in TypeScript. Though the funny flip side is that discriminated union schema type in Zod is possibly one of my least favorite bits of the API. It's something that was submitted as a PR and I accepted. I regret it a little bit. I think it certainly adds value and a lot of people have the need for it.

50:44 My issue with it is you basically, for all the things that you pass in as options to this discriminated union, you need to dive into the internals of those sub-schemas in order to extract out a literal value that is the key. You need to be able to do that for every schema that you pass in, which isn't always easy.

51:12 For instance, here, you're passing in an object into discriminated union, you need to that look at that object, look into the shape, and then pull out the type field. Then you need to make sure that that's a literal or some other kind of Zod schema that just has some actual literal primitive value attached to it.

51:36 What this does is it makes the implementation of discriminated union dependent on all of these internals of the kinds of subclasses that Zod implements. Part of the goal here is that people theoretically could actually create their own subclasses of ZodType if they wanted to and then just define whatever custom parsing logic they want.

52:03 In practice, I don't think people do this, because then, their new subclass isn't reflected in the z.whatever API.

Matt: 52:11 It's this internals coming back to bite you. I'm just reacting to this because this is interesting to me.

52:20 To look at this, this is the Zod.create method for this -- let me open up that playground we had -- where you basically say, this is a good example of some complex types that I just maybe want to end on. We've got this z.DiscriminatedUnion where the discriminator has to extend string.

52:40 Then these types, this is basically an array that has more than one member or at least one member, and then this ZodDiscriminatedUnion option where the discriminator extends string, because this gets to the heart of all of this generic structure, we then have a ZodObject with a map type inside it saying that it has to have the key in discriminator.

53:11 This could be record, probably it is going to break everything, but discriminator ZodTypeAny.

Colin: 53:22 This could work.

Matt: 53:27 I feel nervous coding in Zod. This is not normal, making a PR to Zod with the creator watching right over my phone. This is basically saying this ZodObject has to have the discriminator in it essentially.

Colin: 53:46 Yeah, and this is another scenario where it's nice that ZodObject is this clean abstraction boundary over the base class. All that ZodObject needs, it has other generics of course, but the first one is that shape, which is just an object with keys that correspond to other Zod schemas.

54:11 It's very nice that when you're trying to represent a pretty complicated type here, which is a ZodObject schema that has a shape, that has a property of a given name. That can be represented in this form right here, it's one line. I had to tack on that and ZodRawShape as well to basically have everything work properly, because you need...Are there really no errors there?

Matt: 54:46 That's astonishing.

Colin: 54:50 There might be some test files where issues actually come up if you don't have that, because I'm pretty sure, I wouldn't add that in, unless it was necessary.

Matt: 55:00 I've actually come across this pattern like quite recently. We are running slightly over time. Let me explain what I think this is, which is you use this extra intersection to coerce it to be a certain type to basically say it's almost like an extends in this situation, like a constraint. Am I getting that right?

Matt: 55:27 Yeah, that's a good call because this ZodDiscriminatedUnionOption is used inside of our big inference function here, where we're trying to infer this big array of Zod schema types. We want to make sure that we aren't losing any information on those.

55:45 We need to make sure that this type here is actually going to be generic enough that we can pass in whatever we want while still specific enough that we infer it properly. This is a weird balance that you need to strike when you get deep into the inference weeds of TypeScript.

56:08 If you want interesting little challenge here, try to write a generic function that lets you pass in an arbitrary JSON object and properly infer everything all the way down such that the results of that is exactly equivalent to what you would get if you passed in the input and then used as const.

56:31 As const is a thing in TypeScript that lets you declare these big nested objects containing arrays and primitives. Instead of doing the normal type loosening, where here, it would be A as a number, B as a number, it infers them as literal values. Pretty, pretty...

Matt: 56:52 You'll be glad to hear this is in the first section of the first module or maybe the second section of the first module in total TypeScript.

Colin: 57:01 What am I thinking? Of course, it is. This is a...

Matt: 57:05 Of course, it is.

Colin: 57:06 Yeah, a perfect exercise for people trying to learn this stuff for sure.

Matt: 57:11 Exactly. Why don't we start finishing up there? We've been deep in the weeds. Why don't we pull this out a little bit further back into application code, because this is library code?

57:25 We've seen how complicated things can be where you have generics, which are passed to other generics with various constraints and all this mad stuff. What techniques, what high-level ideas can people pull into their application code when they're thinking about working with generics and working with the advanced stuff?

Colin: 57:50 Starting off, I mean, with understanding how to write a function with generic inputs will get you really far, in terms of what you're trying to represent. It can be tricky when you're trying to interface with other libraries that export their own messy generic types.

58:16 That's a different bit, but I think being able to write a simple generic function and have a grasp of how conditional types works gets you 90 percent of the way there. When you put those two things together, all of these weird messiness in Zod with abstract base classes that are extended by additionally generic subclasses, that stuff is very unlikely that you're going to be able to need to know.

58:46 Just being able to write a generic function, infer the type of whatever you pass into it, and then transform that with the conditional type, this is the TypeScript magic. This is the stuff that TypeScript is uniquely good at doing over other type systems and other static languages.

59:06 Those two things alone, just having some understanding of those, that's really the gist of what you'll need to know to write application code that's properly dry. There is a way to represent whatever you need in TypeScript, like I mentioned, if you sacrifice the dryness a little bit.

59:29 It would be nice if you were able to take some complex type and use all of the TypeScript built-ins or whatever, things like pick and omit in those, to transform from an initial type into a new type that is needed for some scenario. That's nice, it's dry, and lets the types percolate through your code base nicely. You can also redefine. That is the approach I recommend, of course.

59:58 If you're banging your head against the wall trying to figure out some of that stuff, you can always fall back to just re-declaring a few things, be slightly more duplicative. Then just know that the set of stuff that you need to learn from TypeScript, just start wherever you're at, and hen just build up from there in complexity. There's a very natural progression here.

1:00:24 It can be tricky, I think, to really understand why something is important before you in your own life encounter the real use case for it. It's a great feeling when you do encounter that use case, and then you read the module in Total TypeScript that explains it. You're, "Wow, that would be so good for this or that."

Matt: 1:00:43 Yeah, that's my thinking, too. That's a great way of putting it, which is if you want things to be dry in your code base and you have an idea for an abstraction in TypeScript, you're probably going to need a generic function to figure it out.

1:00:58 Colin, thank you so much. That was absolutely illuminating. Thanks for helping me make my first PRs into Zod to remove all the as anys. That's very exciting for me.

Colin: 1:01:09 Good luck with reading all of those. Those were the easy two, I think.

Matt: 1:01:15 Thank you so much, pal.

Colin: 1:01:18 My pleasure. Night, Matt.