Twenty minutes of very confident debugging later: it was the router, and the router had told nobody.

This is not a criticism. I do it too. “The internet is down” is a perfectly good description of the experience. It is a terrible description of the fault, and the gap between those two things is where whole evenings go.

The two questions people conflate

“Can packets leave this machine” and “can this machine turn a name into an address” are different questions with different failure modes. Wi-Fi answers the first. /etc/resolv.conf answers the second. They fail independently, and the symptom — a spinner, then a browser page that says something unhelpful about not being able to connect — is identical.

Worse, the second one fails intermittently in ways that look like the first. A resolver that answers in 40ms for cached names and times out for everything else feels exactly like a flaky radio. It is not a flaky radio. It is a resolver, and the radio is sitting there working perfectly while you power-cycle it.

The network is reliable. The network is not reliable. Both statements are load-bearing, depending on which meeting you’re in.

Four commands, in order

Run them in this order and stop at the first one that fails. The order matters more than the commands do — you could substitute your own favourites for any of these, but you cannot substitute the sequence.1

ping -c2 $(ip route | awk '/default/ {print $3}')   # 1. the gateway
ping -c2 1.1.1.1                                    # 2. routing
dig +short cloudflare.com @1.1.1.1                  # 3. resolution, bypassing local
dig +short cloudflare.com                           # 4. resolution, as configured

Ping the gateway

If this fails, nothing downstream is worth testing. It’s the cable, the radio, or the DHCP lease — and in my experience, in roughly that order of likelihood, which surprises people who expect the radio to be the flakiest part. Cables fail quietly and constantly. Radios fail loudly and then recover.

The useful thing about starting here is that it’s the one test whose failure has an obvious physical response. You can go and look at a cable.

Resolve a name you don’t own

The gap between the third and the fourth command is the whole article. If line three works and line four doesn’t, the network is fine and the resolver is lying to you.

One caveat, which I got wrong in the first version of this piece: on hotel or café Wi-Fi, line three can also fail for a reason that has nothing to do with your resolver. Captive portals routinely block outbound port 53 to anything but their own box, so a failure there means “something is eating DNS” rather than “your resolver is fine”. If you’re on a network you don’t control, read lines three and four together rather than separately.

What to tell the person who asked

Not “it was DNS.” Everyone says that, it has become a joke, and jokes don’t transfer skill. Tell them which of the four lines failed and what that line was asking.

People remember a question far longer than they remember an answer. The next time it breaks, someone who was told “it was DNS” will call you. Someone who was told “line one asks whether this machine can reach the box in the hallway” will run line one themselves, and about a third of the time that’s the end of it.

• Line one fails — it’s physical, or it’s the lease.
• Line two fails — it’s routing, or it’s the ISP.
• Line three fails — something is eating port 53, which on a network you don’t control usually means a captive portal.
• Line four fails alone — it’s the resolver. This is the common one.

That’s it. Four lines, one sequence, and a decent chance you never have to hear the phrase “the internet is down” delivered as a diagnosis again.

1. Yes, you can run all four at once. You will also learn less, and you’ll have to reconstruct the order in your head anyway.

The UniFi controller update reset my DNS filtering profile to “default” without a word. Twenty minutes of very confident debugging later: it was the router, and the router had told nobody.

The venue’s resolver was the whole problem, which I’d written about a month earlier and then walked straight into anyway.

Influenced by my father’s interest (among some other important people) I also got interested in computers and programming which I love working on. Working on the internet on complex distributed systems I started to develop a skill for understanding the interelationships and solving very difficult problems. Much like House MD or the inspiration for that character Sherlock Holmes I found myself highly motivated by the rush of solving those problems. To the point that I sometimes regreted how much I liked it.

I still think my knowledge, skill, and experience doing that sort of work are extensive and I’m starting to think that I’d really like working on the intellectual challenge of creating an objective principle based system of problem solving. Maybe such a thing exists, but I don’t think so I think I would have seen it by now. As a matter of fact I think if it did exist in a completely objective form they’d require it as a basic course in the engineering colleges just like calculus.

I’m not changing my work by any means and have nothing to announce other than I’m going to start to experiment with some research in this area. I’m not fully convinced yet, but I think developing and then communicating something like this could become a central purpose for a decade or three. So I’m going to start looking into it as a sort of Troubleshooting on the Side for a little bit and see what I learn and share along the way.

My real goal here is to try and get some of the connection value I used to get out of Blogs and Twitter way back in the day. I know a ton of people continued writing their blogs and I’m hoping to re-enter that space with my boring takes.

I know this is a big mess, and probably there was no reason to use Angular other than I wanted to learn it. And I know there are broken links and posts everywhere and Blogs are dead, etc, etc. But I don’t care about any of that. I’m enjoying coding a side project. I know I’ll learn from it and I’m enjoying at least the prospect of some connection.

So I moved this off of SquareSpace. They were fine, but I wasn’t motivated to use it. Instead now that I have ownership of so much of the data and machinery in my Rube GoldBlog that I’m excited about it. If you have an interest in telling me I spelled something wrong, or the way I’m using typescript is terrible. (See the code here: mclazarus/logicaldisconnect) or any of the other things you’re dying to correct me on, simply tell me on The FediVerse. I hope to eventually support comments in this angular app, maybe they’ll post to Fedi or something. You can find me here: @mclazarus@jawns.club.

This post is what I call a two-fer. I post something and I also find out of the Drafts Action I wrote to post it works. If so I’ll be able to quick post from Drafts from my phone and that will be super fun.

But it’s much more complex than that isn’t it? When I started working at CoreDial my oldest, Alli, was not even one year old, my second child, Ash, was not yet on the way. I had been married for just two years. Heck I was twenty nine years old. One of the reasons I’ve been drawn to startups and small companies is I enjoy learning, and I learn best and fastest by trying to do things uncertain of how I’ll actually do it. So over sixteen years I learned so much, in so many different ways.

During that time my purpose was to help make CoreDial a success in anyway possible. Since I worked on the technology systems this was mostly making it so things worked, so things could scale, so service would be reliable and predictable. I never thought about what was next or what I was working toward because it was obvious. Our vision was set and adjusted as necessary by Alan, my friend and the company CEO. I just had to figure out how to accomplish certain things, learn, and solve impossible problems. As an engineer these are the things I like doing most.

There were hard times, things I learned about myself, and things that made me hate the job sometimes. But I worked my way through it and found a way to negotiate with those around me to make it what I wanted and still provide value. I was never seriously tempted to leave and work anywhere else. You can tell this by my LinkedIn that is a there as a joke to satisfy some friends who said I needed to have a profile there.

After the purchase I must admit I felt a little lost. I mean the day to day work didn’t change much there were still impossible problems to solve and fun technology to learn but the core purpose was gone. There is a scene in the movie Forrest Gump during the time when Forrest is running back and forth across the US that encapsulates how I felt perfectly. A group of people started following him on his journey and he stops and turns to the crowd and says, “I’m pretty tired. Think I’ll go home now.” The crowd parts to let him walk through and one person yells in frustration, “Now what are we supposed to do?!”

I spent time trying to figure out my purpose and have some ideas and am continuing to this day. I am using ideas and techniques learned from Jean Moroney’s Thinking Lab to help me here. One of the things I’ve learned is that the more complex the work I’m trying to do the more I need to just dive in and try and figure it out as I go. This makes many people I’ve worked with uncomfortable but I’m more uncomfortable trying to rationalize a full detailed plan based on no information learned from trying. In my experience the plan will be proven wrong very early in the work and you will need to adjust. So I go in assuming adjustments will be needed and plan by doing.

This planning by doing is part of what’s going on here. I think in the years of my work I’ve proven myself as a technology troubleshooter. Digging in on impossible problems and finding the answer is something I love doing and I continually try to improve at it. I think something I can do as a 10 year style project is to come up with and write down objective principles of troubleshooting. Helping teach people new skills and ideas is appealing to me and learning how to communicate the ideas makes me learn them even more deeply.

Great so I want to try writing a book on how to troubleshoot. That seems like an objective goal that can be taken on. But how to start?

Well I have to improve my writing and communicaiton which means practice. I also have to explore my ideas and experiment to find good ways to find the principles. Also if I want people to know about it having an audience will be important.

When CoreDial started Twitter didn’t exist, Facebook wasn’t available to the general public, and I had been writing on a blog for around two years and fairly regularly. I enjoyed Twitter at first but I sort of hate it now. With notable exceptions the ideas shared are shallow and the reasoning specious but for some reason alleged pillars of the public discourse take these thought snacks seriously and everyone seems to like to pretend that by arguing here they’re doing something of substance. Also there is a new movement where complex and controversial topics are being “fact checked” by the platform and the government.

So trying to build an audience on the social media platforms to me seems like it would be awful. There is too much temptation to buy into the gaming of the machine to get likes or “eyeballs.” And I want to focus on the ideas.

Additionally part of what I lked about blogging and the web when I first learned about it in the mid 90s was that you could customize these things to be exactly what you wanted. To share the information how you wanted.

So what I’m planning to do is openly build my own personal platform for sharing information about my ideas and evolve it as I go and learn. This will have the added benefit of giving me more technology to troubleshoot, an opportunity to learn even more about modern web and other techonologies, and an excuse to write and share software.

Also I’ll own the output unlike the weird relationship where the social media platforms use your ideas to attract more attention and users and potentially revenue. I’m not opposed to this business model in general but I don’t see how it benefits me over years of playing with it.

So sure I’ll be missing the opportunity to have lots more eyeballs see my information but I think as it grows and becomes useful the people I want to see it will find it and engage with it. I’m not playing a numbers game I’m trying to learn and grow and if by writing these things out I learn then I’ve accomplished my goal. If other people see these ideas and engage and ask questions or point out errors then I’ll benefit even further.

I started this on May 21, 2022 and finished the core writing on May 22, 2022. We’ll see how long it takes before this gets publicly accessible.

Line four fails alone — it’s the resolver. This is the common one.

What actually breaks

Not what you expect. It is never the SMTP daemon. It is DNS records you set up once and never looked at again, a certificate you renewed by hand in a hurry, and the slow, invisible business of your IP address acquiring a reputation you did not ask for.

dig +short MX example.com
dig +short TXT example.com    # SPF
openssl s_client -starttls smtp -connect mail.example.com:587

Run those three every few months and you will catch most of it before someone tells you their reply bounced.

The part I underestimated

Deliverability is not a technical problem, it is a political one, and you are a very small country. The large providers are not obliged to explain themselves to you and generally will not.

I still think it is worth doing, for the same reason I still think it is worth owning the domain your email arrives at. But do it with a second address somewhere else, and do not find out about an outage from your mother.

That probably needs an explanation. Pomodoro is a time-boxing technique that guides you to do 25 minutes of highly focused activity followed by a short 5 minute break. In the PDF the author suggests it should be a real break from any thinking about work. I found the transition from a hard thinking task to a state of mental relaxation to be very difficult to pull off, my mind–already warmed up to the topic–would wander back to the problem I was working on. So I decided to do a set of push-ups to help me relax.

So far I've gotten better at pushups, squats, and sit-ups. Oh, and I'm enjoying my productivity based primarily on the structure to my day. It helps me concretize the idea that I can only do so many things in a day and helps me at the hard task of prioritizing my time. It also helps me measure how long I take at certain tasks. Since I chronically underestimate I welcome the opportunity to improve my judgement.

Recently I unthinkingly upgraded MobileMe to the new Calendar. Which of course broke all of that for various reasons. And this turned out to be the best thing I could do, as now I have a much simpler solution that seems to accomplish the same goals and work more reliably.

Now my phone synchronizes with the Google calendar directly using ActiveSync. And my desktop and laptop BusyCal synchronize with Google calendar directly but only need to be running when I need a calendar there. So MobileMe solved my occasional calendar problems by becoming useless for me.

At this point I'll have to evaluate if there is any point to keeping MobileMe at all or if I can simply use DropBox to keep the other things in sync that MobileMe does for me, and I certainly don't need another email account, or IM account and their iDisk solution is a joke compared to the ease of Dropbox.

Is there any other value to this service I'm missing?

I was glad that I was able to identify the cause of the problem and repair it, in only a few hours.  But I was annoyed that my plans for the evening were shot, and I had an ugly and distorted sense that I'm never able to do the things I really want to do, but must always spend my time responding to emergencies that are my duty.  

I know this view is wrong, but over the prior 9 months or so I'd taken on so much more stuff than I ever had before that I had been wavering between resentment against the world at large and anxiety about not being able to achieve my goals.

I think this experience and my response was a big step on my recovery from going crazy. I reminded myself that to accept those thoughts, about never achieving my goals, would bring about a self-fulfilling prophecy.  And I reminded myself to see how the choices I make even in response to the unexpected emergencies are serving the values I want to achieve and keep.

It was at this time that I also realized I'd gotten impatient. And couldn't remember having that problem since I was a wee tyke when I'd get pissed struggling to untie my shoes.

I think by repeatedly reinforcing a view that I should be getting more done than I was, and actually more than was possible was causing me to have an always present sense of impatience and frustration with myself and others.

I had decided that fulfilling my ambitions should be easy and when I couldn't do it I'd damn the universe for being impossible and myself for not succeeding anyway.

In the end I had to challenge the bad premises by comparing them with reality, and clear out those thoughts that had been gumming up my life.

Oh and if your interested about the AC. I took it apart cleaned a bunch of nasty sludge that had accumulated in the drip pan under the coil. And then using an old wire as a snake, dislodged sludge from the upper part of the pipe. Finally hooking up a shop vac to the end of the condensate line and sucked out anything that would come. It worked fine the rest of the summer. And if there is a next time. First thing I'll do is hook up the shop vac.

So, Anyone have recommendations for companies to clean and service HVAC units annually?

Previously Logical Disconnect served as a diary of rants and various diatribes of no particular purpose and with no cohesive theme. Often it had, as most blogs, come off as complaints and sometimes information that was too personal to be of much use to anyone even those who know me. Primarily it was a personal outlet for unformed notions I wanted to share but with no one in particular.

In some sense the personal nature will continue, however, I have recently decided to use the writing here as a means to further my study of Objectivism.

My plan is to use it to write in order to learn. It became explicit to me when listening to Dr. Leonard Peikoff's lecture series The Philosophy of Education that writing is a tremendously valuable cognitive tool. It forces you to slow down your thought process and make well reasoned arguments. If you don't want the concrete, the writing, to be utter nonsense then you must understand and mentally organize the subject. And by doing this it requires you to integrate the concepts, you wish to write about, into your own hierarchy of knowledge. To really understand the ideas in order to convey them. This is to be part of my defense against any kind of dogmatism or other irrationality in my study of philosophy.

This is analogous to the "trick" I used in college to give focus to study. When it became generally well known that I was doing well, I was often asked for help by other students in my classes. By agreeing to help others I was required to understand the material well enough to convey it to them. And if you have ever tried to teach, especially someone who is honestly seeking to learn, you can't fake this understanding. After spending time teaching the concepts to others, my understanding of the material was unshakable. This activity greatly focused my effort and diminished my need for additional independent study.

By making the writing public I could be subject to criticism or counterarguments which may help me refine my ideas. It will provide additional context to those I have been having frequent philosophic discussions with recently. And will also further my wife's growing understanding that she has chosen a very unique individual to share life with.

I also plan to use the writing here as a tool for introspection and as a secondary value I would like my children to be able to have this as a concrete reference for knowing about their Father. I know reading a copy of a single letter I discovered, which my father wrote years ago at work[1] gave me a different and unexpected perspective and even greater admiration for him.

Yes this is a very selfish motive and personal collection of articles. I intend to derive great value from writing them.

1. Yes letter, not email. I said years ago.

Anyway, the way you get it to work is download the crippled trial, then it uses the magic of your MSN account to know that it is authorized and downloads the parts it needs to uncripple itself. Fine. Except when a couple months later your daughter spills water on the laptop, fries a memory chip, and eventually you end up getting the machine working again, but somehow you unintentionally uninstalled MS Money during the fracas.

So I figure, fine I'll reinstall... from what? Well I guess I download the trial again and it will uncripple itself automagically. No Dice. It has no idea I should have the paid version. So I start trying to figure out how to contact MS to fix it. Seems I would have to register a credit card to call them or email their support and risk them charging me at least $100 if it turns out I am too stupid to find the checkbox that says it should be uncrippled. This mental block is enough to keep me from doing it for a while.

Today I vow to do something about it, and I [look](http://www.roosster.com/rstr/) and see that I can buy MS Money 2006 Deluxe from Staples and if I buy Taxcut, I can get some rebate stuff. Well I don't really want taxcut, but I see that I can get a $40 rebate, and I also find that Target has Taxcut Deluxe+State for $20, and MS Money 2006 Deluxe for $20. And the rebate in question is offered from HR Block. Therefore, I run to Target. Pay $42. And I will mail in a rebate form and get a $40 check in a couple weeks. Sure I am out some effort and $2. But the risk of paying potentially $100, and the added bonus of having an actual uncrippled CD to reinstall the software as I like outweighs that.

So I think the moral of this story is: * Don't buy mass marketed stuff directly from the manufacturer because a better deal can surely be found from the purchasing power of a retail outlet. * Don't buy downloadable software if you can get a hard copy of it for the same or less cost. * If you wait until January you can most likely upgrade financial software for free.

Basically I start trying to plan out some structure that is used to capture the main things I need capturing, which anyone can tell you are a list of your projects. A list of your next actions. And some sort of calendar to capture the hard landscape of what is happening. And finally a reliable reference file. Where to store stuff you may need to reference later, reliably, in a form that is easy to look up.

Well as a guy who builds systems and programs this is a nightmare, I start thinking this is simple enough to build something. Well it isn't, because all of the sudden I have something way more complex than the [37signals](http://37signals.com/) people have built, and it will take me a good 8 or 40 hours to put together. And then I start thinking about backup, because hell if this is my life I better be able to back it up. And of course I don't have time for this because while I am thinking about this I am shirking all kinds of other responsibilities, and the fires are growing all around me. Then I run around for 3 days putting them out and long for an uninterrupted month where I can construct this super-duper life runner application.

Well I am a father, and I work, and I have hobbies, there is no uninterrupted month coming any time in the forseeable future, which I can't really see much of anyway because I am so busy running around putting out fires, or planning for the utopia in some distant future. I really need to work on my middle game.

Anyway, starting today, I am going to errect a little bit of scaffolding to try and hold my life together. I am going to spend about 10 minutes thinking about the simplest system that can possibly work.

*I am picturing a calendar and a notebook.*

I think the biggest thing here is consistency of putting everything there, and to get in the habit of looking at it several times a day. Maybe I can trick myself and write "email" or "bloglines" on the cover, since I typically look at both of those several times an hour.

I have to convince my brain that it doesn't have to be perfect to start using it.

Basically it involved taking dow the vinyl cap that was covering the underside of the eaves, and snapping 2 chalk lines 2 inches apart, using my new circular saw (got it for $45 on clearance at Lowe's) to cut a hole, affixing the vents and replacing the cap. It gave me a chance to clean up all the dirt and caccons and crap that were on the hidden side of the caps, and I noted that some previous owner had 24, 2 inch diameter holes drilled into the soffit to provide some ventilation. Of course they didn't put any kind of vent plug in there, so I bought some of them and popped them in also. And I am pretty sure the free space provided by the holes was not sufficient.

To get the how-to on doing this job I searched google for soffit vent installation. And read the first 20 or so pages returned. I basically followed the instructions from here. Now the only concern I have is that the vinyl caps aren't allowing enough airflow, in that case I will have to find new caps that are more air permissive. And go through the pain in the ass job of cutting it to fit.

Hopefully it isn't raining on Saturday, and I will be able to do the back and finish the job.

By the way, I also purchased a Canon PowerShot S410 camera this weekend. It rocks! The pictures above and other pictures of the job were taken with it, and can be found in my gallery. However, there is no way this expensive piece of equipment will be capturing the glory at the bike race, that is a job strictly for the much more inexpensive pen-cam.

A company I am associated with has a Cisco Pix 515 in their main office and actually happens to have a spare 501. So they thought it would be a good idea to have a VPN between their main office and their branch office. Their branch office has a Lucent Pipeline 75 for their 128 Kbps ISDN connection. The offered me a nice payday to get the VPN to work for them, so having quite a few years of Cisco and general network experience I happily accepted. Read on for a technical overview of the problems and solutions I encountered.

The initial network I found in the branch office was quite simple. A static IP address on the Pipeline that was configured for Port Address Translation (PAT), or in Lucent Pipeline terms, NAT with a single address. Probably a pretty common configuration for an office like this and most any house network using one of those Linksys now Cisco devices.

The problems began when I realized the NAT/PAT on the Pipe75 wasn't smart enough to do some sort of workaround translation on ipsec packets. The incompatability and workarounds on this are generally well known. Googling for the various combinations of vpn, ipsec, nat, masquerading, etc. will give you some nice background material on the problem.

Well now I run into the first major roadblock and that is that Lucent is terrible at providing information and support on the products the assimilated. I don't have a support contract with Lucent and neither does this company, they just were given the hardware when they signed up for ISDN service many years ago. In order to get access to Lucent's knowledge base and forums apparently you need to pay them for support. Obviously this kills of any secondary free support structures that could arise in their forums before it even could begin. Well I was able to locate some useful documents: Pipeline 75 Users Guide and the Pipeline 75 Reference Guide. Also I was able to get a support account that had minimal access to some whitepapers that Ascend published for the customers years ago, but none of them dealt with this situation, only with trying to get the VPN implementation on the Pipe75 to work.

I was able to get the ISP to issue an additional Static address. So now I have to /32's to work with. The pipe75 and the Pix. Now what I am going to do leave the original static address on the external interface of the Pipe75, I am having the ISP statically route the other slash-32 to the first one (the external pipe75 interface).

And here is where I got a little ghetto or clever, depending on how you look at it. Say the addresses I have are 172.16.1.105/32 and 172.16.1.180/32. Well 172.16.1.105 is the outside interface of the pipe75, and somewhere the ISP should have an entry like: ip route 172.16.1.180 255.255.255.255 172.16.1.105. Now the internal interface of the pipe75 is set to 172.16.1.177/29 and I set the external interface of the Pix to 172.16.1.180/29, and I set the default gateway on the PIX to go to 172.16.1.177. Yeah I am masking out a few real addresses. But it won't effect the ISP and I am severely doubting that the users at the branch office have any reason to go to those real IP addresses I am masking out. Well anyway it is a risk I am willing to take. So at this point all I need to do is turn off NAT on the Pipe75 and I am good to go.

The status at this point is I am waiting on someone at the ISP to enter the route for me. Then I can get into the Pipe75 and turn off NAT. The Pix and everything else are already in place as they should be, just need that route so I can stop doing uneccesary translations which are breaking my ipsec, and all other non-ipsec traffic will just be PAT'd by the PIX to 172.16.1.180.

If nothing else, I have some nice notes here, if I ever need to revisit this configuration.