Filed under: Work | bbot's blog

bbot's adventures in load shedding

bbot — 2008-08-18T05:31:27-05:00

So I was changing lightbulbs on floor 26, which is entirely taken up by offices of the Washington State Court of Appeals, whose court is on the lobby floor. It had apparently been a while since they had been graced by the presence of a lighting maintence technician, since I was seeing a ton of light bulbs, and people kept leaving post-it notes on my clipboard, asking me to change some particular fixture. I'm doing the whole floor, guys, you don't have to draw my attention to every single fixture with a dead bulb.

When but should we recieve an annoucement over the security intercom that PSE had experienced a series of transformer failures, and that we urgently needed to reduce power consumption by 40% or suffer a brownout. To wit: we would shut off all the lights. (Except for the emergency lights, which cannot be shut off, except via axe.)

Now, you would think that I, being a lighting maintence guy, and thus not an actual electrician, would have no duty to play in the "shutting off lights" phase, and, as a corrolary, would not be able to fufill my normal duties, since changing lights is hard if you have no way to determine which ones are dead. You would be wrong, because as a Union Square employee, I did have a valuable role to play in the "intimidating tennants" phase.

You see, the lights are shut off through the building automation system, which turns the lights on in the morning and turns them off at night. For the purposes of working late, an override switch is provided, to override the automatic shutoff and keep the lights on.

But this switch harbors a sinister secret, for it not only overrides the timers, but all commands from the management computer; say, if it was attempting to shut off the lights to prevent a brownout.

My job was thus: Track down these people, and gently point out that we had a contractual obligation to the city of seattle to shed load, and if they kept overriding management commands, we would unfortunately be forced to shut down all power to the floor; which included power to computers, critical servers, life-support machines, (also known as "coffee machines") etc.

This had the unfortunate side-effect of making me feel like a jackbooted thug.

After properly browbeating the offending tennats, I proceded to the One Union engineering office, where Don Wildes[*] and Mitchell (coming to us live by speakerphone) were manning the building management computer. I fielded calls on Don's phone while he resoved some alerts, then we proceded to 17 to clear a taped override switch.

*: My immediate supervisor for the filter changing and relief fan maintence project of 12/2007-03/2008.

Now, to me, the procedure to clear a taped override switch[**] is:

1. Remove the tape.
2. Leave.

But Don switched it up, and added a third item:

3. Stick around and tell the tennant how walking into their suite and shutting off their lights is good for them.

**: If you tape an override switch in the "on" position, then it'll constantly increment the timeout counter, and the lights will, as a result, never turn off. This is a fairly bad idea, since it means that the lights will never turn off, and thus, you know, remain on. So, removing the tape benefitted the tennant in three ways: It prevented a brownout, it saved them money on their power bill, and it saved them money on lamp replacement costs.

The person closest to the door took this as an oppratunity to point out that it was dark, that he didn't like it being so dark, and that he would rather it be brighter. At one point in his increasingly hostile rant he speculated that it "was bad engineering" for there not to be one light fixture per office. He, of course, couldn't know this, but Don is an Operating Engineer, and this is more or less a direct insult.

Don, remaining remarkably calm, replied that he should contact the architects who designed the building in 1982, and express his concerns to them, rather than the two people with the least possible ability to affect the situation. In response, he told Don that he was "tired of your sanctimonious shit", and told us to leave, which we did.

Nothing much of interest happened after that. The "crisis" ended a couple of hours later, just as I was leaving, and Mark got to turn all the lights back on. EXCITING

another day in the life

bbot — 2008-08-09T07:42:28-05:00

So what happened this week? Not much!

Wednesday: My eee pc arrived, and oh my god it is just as cute as hell. Reading about it on the internet failed to convey its size, so here's some size comparisons, using random crap from around the office. (water bottle, two gig ipod nano)

And one with one of my enormous mitts covering the keyboard.

It's friggin tiny! The ergonomics are a predictable trainwreck, and it sure ain't fast, but what do you want for three hundred and ninety nine US dollars?

There was a bum violently objecting to be thrown out of the building just outside of my office today. It was a nice way to start the day.

Also, I shocked myself for the first time at work! I was replacing a ballast (for the second time, since the first one was DOA) and I brushed up against the hot wire.

Q: Why didn't you cap it off, jerkface?

A: I did, assface, but the wirenut fell off.

We use 277 volts for the lighting circuits, so it was pretty spectacular. I let out a kind of "Augh ugh ugh ugh" noise and stumbled off the ladder. I, of course, had a witness, which was great. I taped that son of a bitch wire nut back on, which was somewhat unwise, in retrospect, since I only had one more connection to make before bringing the fixture back online; but I would be damnned if I let that little bastard shock me again.

Nothing else of interest, except that the 15:40 301 was both unusually early (by a full three minutes) and unusually uncrowded.

You want more Failtrain drama? Heck yes you want more failtrain drama!

Kyonko defends himself against my vile accusations.

what was this one man attack on tf2chan
what is this
and when did this happen
i do not remember doing this at all
i remember that lol
i only read the parts that were near my name
when they didn't show up for the scrim, and you micspammed their server
it was the day we had the scrim against mantrain
well
i chatspammed
you went to their server and ran around
and they got pissed, and saz had to talk them around
and yeah
did that
and they left after a few rounds anyway
i just remember chatspamming
yeah it was pretty lame
i don't remember what else you guys are talking about
that is perhaps due to your exceptional stupidity
or

[time passes]

nice rebuttal
i know right

What happens when you shut down a server in response to widespread abuses of power? Why, people copy you. And then they impersonate you. For the record, my IP is 71.112.185.202, and I still don't know the rcon password. That didn't stop Dr. O, however. The logs are of #tf2, which is linked to the failtrain via FailtrainRelay. Comments to be relayed to the server are prefaced with !failbot.msg.

19:52 <%FailtrainRelay> (| *TKM* | Dr. O): you gonna write about it in your blog some more bbot?
19:52 < bbot> !failbot.msg WILL DO
19:53 <%FailtrainRelay> ([TGIF] Balki Bartokomous): do you have a journal?
19:53 <%FailtrainRelay> ([TGIF] Balki Bartokomous): actually
19:53 <%FailtrainRelay> (jclark): dammit, I don't get uber from you if you god mode :(
19:53 <%FailtrainRelay> ([TGIF] Balki Bartokomous): a diary seems more appropriate
19:53 <%FailtrainRelay> (| *TKM* | Dr. O): bbot does
19:53 <%FailtrainRelay> (Scruffy): !slay kermit
19:53 < bbot> !failbot.msg bbot.org/blog/
19:53 < bbot> !failbot.msg quick, alt+tab out of tf2 and go there
19:53 < bbot> !failbot.msg or, you know, not
19:54 <%FailtrainRelay> (shanitos): SPIES EVERYWHERE
19:54 <%FailtrainRelay> (OhshiAppledrink): !slay Dr. O
19:54 <%FailtrainRelay> (OhshiAppledrink): !slay Kermit
19:55 <%FailtrainRelay> (shanitos): noscope'd
19:55 < bbot> !failbot.msg !slay !@me
19:55 <%FailtrainRelay> (Ambassador Dick von Butticus): !slay some
19:55 <%FailtrainRelay> (| *TKM* | Dr. O): !slay shan
19:55 <%FailtrainRelay> (shanitos): aww
19:55 <%FailtrainRelay> ([TGIF] Balki Bartokomous): GO SAZ YOU CAN DO IT
19:55 <%FailtrainRelay> (Tach'ko): Wait for it...
19:55 <%FailtrainRelay> (King of the Internet): timeleft
19:55 <%FailtrainRelay> (Tach'ko): Got 'em
19:55 <%FailtrainRelay> (King of the Internet): nextmap
19:56 <%FailtrainRelay> (俞坤霞Eva=HKC=): lol
19:56 <%FailtrainRelay> (King of the Internet): !god kermit 0
19:56 <%FailtrainRelay> (OhshiAppledrink): !slay Saz
19:56 <%FailtrainRelay> (Scruffy): /godmode scr
19:56 <%FailtrainRelay> (Sazpaimon): !god @all 0
19:56 <%FailtrainRelay> (King of the Internet): !god @all 0
19:56 <%FailtrainRelay> (King of the Internet): !god @all 0
19:56 <%FailtrainRelay> (shanitos): ;_;
19:56 <%FailtrainRelay> (jclark): dammit, you wait until I die
19:56 <%FailtrainRelay> (King of the Internet): !slay @blue
19:56 <%FailtrainRelay> (Ambassador Dick von Butticus): doh
19:56 <%FailtrainRelay> (MajorMcOaf): lol
19:57 <%FailtrainRelay> (Justin): !noclip @all
19:57 <%FailtrainRelay> (shanitos): OH GOD
19:57 <%FailtrainRelay> (OhshiAppledrink): !unfreeze Apple
19:57 <%FailtrainRelay> (俞坤霞Eva=HKC=): D:
19:57 <%FailtrainRelay> (King of the Internet): !god king 1
19:57 <%FailtrainRelay> (| *TKM* | Dr. O): !kick @!me
19:58 <%FailtrainRelay> (俞坤霞Eva=HKC=): hurrr
19:58 -!- Scientus [chatzilla@nin.ja] has joined #tf2
19:58 <%FailtrainRelay> (shanitos): :C
19:58 <%FailtrainRelay> ([TGIF]Kermit The Frog): GO MAN, GO
19:58 <%FailtrainRelay> (King of the Internet): !god @blue 0
19:59 <%FailtrainRelay> (King of the Internet): !slay dr.
19:59 <%FailtrainRelay> (shanitos): D:
19:59 <%FailtrainRelay> (shanitos): ADMINS = FAGS
19:59 <%FailtrainRelay> (Scruffy): /slap @all 50
19:59 <%FailtrainRelay> (OhshiAppledrink): !slay Dr. O
20:00 <%FailtrainRelay> (| *TKM* | Dr. O): !ban tach'ko 30 existing
20:00 <%FailtrainRelay> (俞坤霞Eva=HKC=): D:
20:00 <%FailtrainRelay> (shanitos): BAWWW
20:00 <%FailtrainRelay> (俞坤霞Eva=HKC=): oh god
20:00 <%FailtrainRelay> (Justin): sm_rcon quit hurrrrrrrrrrrrr
20:00 <@Scruffy> /r/ removal of dr. o's cheat commands
20:01 <%FailtrainRelay> (shanitos): dohohoho
20:01 <%FailtrainRelay> (bbot): YOU GUISE R MASSIVE FAGGOTS!!! I HAET YOU ALL!!
20:01 -!- FailtrainRelay [failbot@muffins.rawr-A915721F.onlinehome-server.com] has quit [Client exited]
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:01 < bbot> !failbot.msg oh god I'm in two places at once
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:01 <@Scruffy> RAGE SERVER SHUTOFF
20:02 <@Scruffy> RAGE SERVER SHUTOFF
20:02 <@Scruffy> RAGE SERVER SHUTOFF
20:02 <+Sazpaimon/cultnet> that was great
20:02 < bbot> fucking saz inpersonating me
20:02 < bbot> er
20:02 < bbot> impersonating?
20:02 <%RMC> Wage souvah shutowff
20:02 -!- jclark [Josh_Clark@muffins.rawr-15317BAF.dynamic.ip.windstream.net] has joined #tf2
20:02 <@Scruffy> and it was dr o
20:02 <@Scruffy> not saz
20:02 <+Sazpaimon/cultnet> even without access to the server
20:02 < jclark> hurfa durf
20:02 <+Sazpaimon/cultnet> rcon from "24.240.93.5:3730": command "exit"
20:03 <@Scruffy> not my address
20:03 -!- FailtrainRelay [failbot@muffins.rawr-A915721F.onlinehome-server.com] has joined #tf2
20:03 -!- ServerMode/#tf2 [+h FailtrainRelay] by cultnet.janus
20:03 -!- DocO [Doctor@cult-ac7fcc0a.wi.charter.com] has joined #tf2
20:03 -!- ServerMode/#tf2 [+h DocO] by cultnet.janus
20:03 <%DocO> yea
20:03 <%DocO> YEA
20:03 <%DocO> YEA
20:03 <%DocO> YEA
20:03 <%DocO> YEA
20:03 <%DocO> FUCK YEA
20:03 -!- mode/#tf2 [+b *!*@*.bbot] by DocO
20:03 -!- mode/#tf2 [-b *!*@*.bbot] by Scruffy
20:03 < bbot> lol wut
20:03 <%DocO> ;_;
20:03 <+Sazpaimon/cultnet> that was the best 10 minutes of failtrain
20:03 <+Sazpaimon/cultnet> ever
20:03 <@Scruffy> yes
20:03 <%DocO> bbot why'd you crash the server?
20:03 -!- mode/#tf2 [+oooooooooooo DocO FailtrainRelay RMC Sazpaimon/cultnet bbot coleco Cryomancer/cultnet janus jclark K2 MajorMcOaf Ovolbrod] by Nikki
20:03 -!- mode/#tf2 [+oooo poe_ QuiQafooQey Quisty Scientus] by Nikki
20:03 -!- mode/#tf2 [-o+b bbot *!*@*.bbot] by DocO
20:03 -!- bbot was kicked from #tf2 by DocO [DocO]
--- Log closed Tue Aug 05 20:03:55 2008
--- Log opened Tue Aug 05 20:04:48 2008
20:04 -!- bbot [bbot@bbot.bbot] has joined #tf2
20:04 -!- Irssi: #tf2: Total of 17 nicks [5 ops, 1 halfops, 0 voices, 11 normal]
20:04 <%DocO> FUCK YOU RAXO
20:04 -!- Irssi: Join to #tf2 was synced in 3 secs
20:04 -!- mode/#tf2 [+b *!*@*.bbot] by DocO
20:04 -!- mode/#tf2 [-b *!*@*.bbot] by Scruffy
20:04 < bbot> I didn't crash the damn server, I'm still banned.
20:04 -!- mode/#tf2 [+b *!*@*.bbot] by DocO
20:04 < Vu> so
20:05 < Vu> whats going on
20:05 < Vu> over here
20:05 -!- bbot was kicked from #tf2 by DocO [DocO]

wherein Fads complicate my Job

bbot — 2008-07-26T08:13:48-05:00

At work we've been going through Leadership in Energy and Environmental Design (LEED), because it's not a corporate fad without a certification process to cash in on it. And I do mean cash in, dues are $5,000, and the various fees are charges are also measured in kilobucks. We've budgeted $50,000 for the certification process, because this is obviously far more important than a new HVAC automation system or upgrading the elevator control system. (Entrapments, which is the technical term for "elevator doesn't let the people inside, out", have tripled in the last six months. Which, you know, is bad.)

Anyway, as part of the LEED cert process, we're been moving to more efficient and lower mercury lamps. The 50R20 spotlights in the elevators have been replaced with 45R20s, (First number is wattage, second is base size, letter is base type) F8T5s are being replaced with some low mercury lamp, and the building standard bulb, the four foot FO32T8/730/ECO, has been replaced with the FO28T8/830XP/SS/ECO3. [239KiB .pdf] (Consumes four less watts, delivers the same light output, has superior color rendering and hardly any mercury; and shits unicorn giggles. It also costs four times as much as the FO32, but hey, I'm not paying for them.)

And that's the rub. You see, I use a lot of four foot T8's. As in, sixty or ninety tubes a week. So it's only reasonable that as part of a big Going Green initiative, changing a big thing like the building standard light bulb, you would first check that our supplier had plenty of it in stock. Right?

I placed an order for 240 bulbs on the 26th, and 140 of them were delivered on the 24th. Of July, twenty-eight friggin days later. And it wasn't even complete! It's kinda hard to change bulbs when I don't have any bulbs to change them with, guys!

But now I've got them, and everything's good! Except I blew through two cases doing all the stuff I couldn't do until I had light bulbs, which leaves me with not a lot of bulbs. And judging from their performance to date, I'll be waiting another month to get the next shipment in. And I'm running out of 45R20s, and FBO32/730s, and CF13DS/835s...

All I got is problems!

a day in the life

bbot — 2008-07-09T03:49:36-05:00

I got to use my volt tester twice today, which is good, because I carry it around all day in my pocket protector; but bad, because I only use it if I'm not sure an action that I am about to take will kill me or not.

I replaced a socket in a fluoro can light; because when one of the lights burn out, in this particular type of fixture, the other socket then gets hot enough to melt to the folded tube's plastic base, but not hot enough to trip the thermal protection fuse. When the second lamp dies of accelerated old age[1], you get up there (on a 10-foot ladder, of course) to discover that the lamp can't be changed without pulling the entire fixture apart and replacing the socket.

All this means that every time I walk through the lobby I have to stare up at ceilings, hoping to spot a light out. I can't just look for dark cans, since there's two bulbs per fixture. Thanks, whoever designed this bastard. Thanks a lot.

However, replacing the socket requires shutting off power at the panel, since there's no service disconnect inside the fixture (not enough room) nor on the junction box feeding the fixture. (prohibitively expensive) However, again, this means shutting off the lights in a elevator lobby, so repairing the fixture is best done with haste.

Guess who didn't manage to repair it with haste? Me.

Vlad had been sitting on this fixture for a couple of weeks, hoping that nobody would notice that it was only holding one bulb, but eventually that bulb went out, (Two days after he retired, of course.) and I had to go up and change it. He had told me, quite clearly[2], which breaker controlled that bank of lights, so I switched it off, and screwed on the fiddly little circuit breaker lockout to ensure that nobody would flip it back on while I was at the top of a ladder[3]. Walk back out to the lobby, and discover that, nope, that's the lobby entrance lights, and hey, wow, looks like emergency exit lights start blinking when you cut off their power!

So I start flipping each breaker with the word "lights" in its tag.

Breaker 2? 6? 11? 31?

Ah, that's the ticket.

Replacing the socket was uneventful, (checking it with the volt tester to make sure it was off, returning to the topic) but turning the power back on revealed that shutting the power off at the breaker was traumatic enough to kill lights in two other fixtures.

The second time I used the volt tester was while replacing a ballast. The work order had been for a light out in Reception, and in the employee kitchen. The reception bulb was a weirdo, a BR base (incandescent floodlight form factor) compact fluorescent; (model number CF16BR30, in case you care) but in the kitchen, all four lights in the fixture were out, which is pretty damn rare. Engaging in some forensics while waiting to see if the lights would burn out[4], just working off the dates, it looked like three of the lights went out, and the fixture limped along with one for a couple of years (!) before the last one went the way of its brothers.

On my way out, I noticed some lights out in the secretarial quad, and after getting authorization,[5] I started changing them. One fixture with both lights out did have a bad ballast, (which required using the volt tester for the second time) and by the time I left I had gone through 12 tubes, about half a case.

After dropping my cart off at my office, I went down to the loading dock to get some of those chocolate wafer things, when I discovered that we were in the process of having a new vending machine delivered. The guy had just finished testing it, so I offered to be the first customer.

The vending machine had an XY (rather, YZ, since it didn't traverse the depth of the machine, ah ha ha ha) gantry, with a basket and a motor on it. The gantry would move over whatever row you selected, the motor would engage a sprocket on the front of the row, and vend whatever you selected. I put my money in, it goes through its song and dance, and nothing happens. The gantry resets and repositions itself, nothing happens. Gantry moves back to the bottom row, and now the display reads "SELECT ANOTHER PRODUCT".

Vending machine guy: Huh.
Me: At least it didn't steal my money.

Some troubleshooting later, we figure out that the row itself wasn't positioned correctly. Pop it open, fix it, now it vends correctly. Get back to my office, and the mountain dew's warm, since he had just put it in.

ALL I GOT IS PROBLEMS

1: The typical end-of-life failure mode of a fluorescent lamp is failure to achieve discharge due to insufficient gas ionization, which is in turn due to cathode failure, high gas pressure, or incorrect gas mixture.

The electrode filaments on fluorescent tubes are great big beefy things, especially compared to the ones in incandescent bulbs, which have to be thin thanks to physics; but the therminonic emission "catalyst" (not literally a catalyst, since there are no chemical reactions occurring) coating on the cathode filaments still boil off fairly briskly. Some of these newly liberated ions will slam right into the side of the tube, producing the dark spots on the ends of old fluoro tubes, and, eventually, cracking them; allowing the atmosphere in, which both changes the composition of the gas, and raises the pressure high enough to extinguish the discharge, but the rest of the ions are lost forever.

This is bad, because the coating is what enables the discharge, and when enough of it boils off, the lamp won't light, or, in borderline cases, will only light when cold. This causes the light to turn itself on and off (the electronics will manage to light the lamp, but internal pressure will rise as the lamp warms up until the electronics can't supply enough current to sustain the discharge, at which point the lamp turns off, rinse, repeat) until the temperature cycling boils enough catalyst off to kill the lamp dead. This doesn't occur very often in fluoro tubes, since they start so quickly; but is fairly easy to observe in aged low pressure sodium bulbs; since they warm up slowly, thanks to their greater mass.

The point being, heat kills fluorescents, and keeping them hot kills them faster.

2: Lit. "Looby lights, they are controlled by breaker von on panel seeven. I am Russian, despite how bahdly Sam is mangling my accent!"

3: Being shocked at the top of a ladder usually results in death of falling off a ladder.

4: In almost every case, an entire fixture going out means the ballast's bad. So far, I've only seen this failure mode twice.

5: Some tennants like to leave fixtures half-populated, to save energy, and to confuse poor lighting fixture maintenance personnel. On these occasions, I only change lights when directly asked by the person sitting closest to the fixture.

employed! III

bbot — 2008-06-17T04:31:53-05:00

As of today, I am employed at Union Square as a Lighting Maintenance Technician.

It's interesting working at union square again. I'm seeing all the people who I saw going home at the start of my shift last time, but all the time. It's weird! And easy to put into words, as you can tell by that masterful sentence there.

Another item of note, this is my first full time job that actually takes place during normal waking hours. Pro: Daylight; other, somewhat more normal, people. Con: Traffic! Now I know why everyone constantly complains about Seattle traffic. I don't actually have to drive in it, but the buses are absolutely packed.

unemoloyed! II

bbot — 2008-03-27T09:34:43-05:00

As of 02008/03/26, I am no longer employed at Union Square. My contract ran out, and management elected to not renew it.

leak testing

bbot — 2008-03-02T06:32:59-05:00

Lately I've been checking for leaks. How does one do this, you ask? Easy!

Check system pressure at the EP box, where each floor taps compressed air from the central air line. 19 times out of 20, pressure will be nominal, maybe a pound below central pressure.

One time out of twenty, though, system pressure will be outside of tolerances, and you have to proceed to the actual leak checking phase of the operation.

Now, at two union, the system air is piped to a loop that circles the floor, with taps where perimeter fans and thermostats draw air from it. Everything is operated by air, of course, because two union was built in 83, before the invention of electricity.

To test for leaks, you cut the loop right by where central air is connected to the loop, but before it is tapped to power a thermostat/perimeter box, since this would isolate it from the loop, and thus render it nonfunctional. This isn't bad, in the sense that this is done all after-hours, and thus with nobody to complain about the heating suddenly not working, but bad in the sense that you can't test a box for leaks if it's not connected to the loop.

Anyway, once you cut the loop, you go halfway around the floor and connect a gauge to the line. You then squeeze shut the line directly after the gauge, and if system pressure goes up, you know that the leak is somewhere after farther down the loop. If it doesn't go up, then the leak is farther up the loop. By doing this over and over, you can narrow down the location of the leak enough that you can just trace the line by popping up ceiling tiles, which would otherwise take far too much time.

This is idealized scenario, of course. What actually happens is that when you plumb in the gauge halfway around the floor it will read one PSI, stubbornly refuse to budge from that reading no matter where you clamp the line, and then strongly insinuate unkind notions regarding your mother's reputation.

Dejected, you will wander the floor at random, immediately stumble upon the leak, (a contractor cut a branch but was too lazy to plug it) fix it, and hurry to your blog to tell the world about how awesome you are.