Symptoms

Possible Causes

Solution

Can't lock the gunn

Mistook the gunn backshort as the tuning micron--a well known side effect of oxygen deprivation.

Set the gunn tuning micron and backshort to the proper value prescribed per the tuning table on the sidecab wall.

YIG not locked

Flip the local/remote switch on the PLL/YIG box to local and back to remote. Check if the YIG green LED indicator is on.

Gunn in protection mode as indicated by a fully deflected voltmeter on the PLL/YIG box

Load a different line under UIP (preferably using a different receiver), and reload your line.

Receiver draws little current (<5 mA)

Mixer attenuator not open or not close to fully open

Open up the mixer attenuator (clockwise)

Multiplier setting not optimized

Fiddle with the multiplier till you "peaked"

Wrong gunn or wrong multiplier

Replace with the right one if you know what you're doing. We'd prefer that you let us know and we'll put it on on next working day.

Unusually high system temperature

Hot load left in the beam

Don't kick yourself. It's oxygen, or the lack of it. Go quietly to the sidecab and remove the hot load from the beam.

Sidecab mirror not in place

Reload your line under UIP with /force option and answer yes when asked if you want to rotate the sidecab mirror. Make sure antenna is at a ZA <45 degree before you do this.

Tertiary mirror not in place

Go out to the pedestal and check if the light path coming down from the secondary is diverted to the sidecab. If not, you'll need to get underneath there and lock the tertiary mirror in place (finger-tight will do).

Locked in wrong frequency (5th harmonic in the 345 GHz receiver)

Retune the receiver to avoid the 5th harmonics.

The image sideband covers frequencies of low atmospheric transmission due to improper positioning of your program line

Redefine your program line with "forget 'your_line'" and "line ..." under UIP and placing it in the other side band. Or just reload the line with "lo 'your_line'/upper (or /lower)", forcing the line to be placed in the appropriate sideband.

Sidecab mirrors misaligned

In some rare occasions, sidecab mirrors (4th or 5th, for example) might be misaligned. You'll know this if you've exhausted all other known causes of high system temperature. To realign the sidecab mirrors, you would need to log onto the antenna computer as user cso (look around in the control room for the posted password), and run a script called sidecab.home. Read this for the latest procedure regarding homing sidecab mirrors. After homing the mirror, you can load your line under UIP. The mirrors shuld be in place.

Bad weather

Well, we can't help you with that :)

Backend/AOS Related

Ripples that won't go away

Your scan is taken too soon after a CAL scan

Wait for another 20-30 sec and repeat your scan

You have taken CAL scan too frequently

In most cases there is no good reason to take a CAL scan for every or every few on-source scans. You'll find yourself being rewarded handsomely with better spectra and more time on source for doing CAL less frequently.

Backend computer/AOSes are playing up

Restart AOSes you're using by "aos/restart" under UIP. If that doesn't fix it, reboot the backend computer then restart AOSes..

Integration cycle hangs

This could be any number of things... Let's start with antenna computer/backend computer/AOSes first.

If antenna is tracking and updating status properly, check if the backend is alive (ping bck). If both are okay, kill AOSes and downconverter related processes on alpha first (stop/id=proc#, where proc# is found by "show system" on alpha. Then you are ready to restart AOSes under UIP. If this doesn't fix the problem, reboot the backend. You might as well reload the antenna computer with ant/restart/nosync under UIP in the mean time.

One other possibility is that alpha1 crashed, which rarely happens these days. You'll know this if your UIP session on alpha1 freezes. Yet another possibilities is that USER disk on alpha1 freezes for some reason. Scans would not get stored as a result.

Reboot alpha1 by pressing the button near the lower right corner on the back of the alpha1 box. Since we have three nodes in the VMS cluster, the two others (poliahu and alpha2) will remain functional (to certain extends) while alpha1 is booting. Don't take it too liberal though to boot alpha1as the mirrored the user disks go through a lot of threshing to copy each other each time alpha1 is rebooted.

One other little inconvinience is that you won't be able to access your VAX directory from the SUN workstations until the user disks finished copying each other AND you restart the NFS server on alpha1 by the following:

ALPHA1::smith>multinet config/nfs

NFS-CONFIG>restart

NFS-CONFIG>exit

It'll tell you if you don't have sufficient previledge. If you don't know the password to become system to do this, page your friendly CSO staff member on call for the night for the password. Or if you don't feel comfortable rebooting alpha1 and restarting the NFS server, page the on-call person and have him/her walk you over the above procedure.

Yet another possibility is that knoppix, the linux box dedicated to the X display of AOSes by default, is hung for some reason. We have yet to see this happen once over six months or so since knoppix was switched on. But we're here covering the hupothetical grounds...

Reboot knoppix by pressing the big power button on the front of the box. When presented with the Boot: prompt, type "knoppix noicon" followed by a carriage return. The box will come up with a cute Tux staring at you at the center of the screen. You're ready to restart AOSes under UIP.

Calibration scan hangs. This can manifest itself as a hung OTF scan, as a Cal scan is done by default at the beginning of a row in the OTF scan.

Again, a number of things could cause a cal scan hang. backend computer crash or antenna computer crash are the first ones to check.

make sure backend computer is up and running. See if you can "ping bck", and if the AOS displays are still alive on puuoo. If not , go press the reset button on the backend computer to bring it back on line. For antenna computer, see if you can "ping hau", and if the antenna is still tracking the source and the time is updating on the status monitor. If not, you can reboot it the nice way if you can still "ssh hau -l cso", or the rough way by pressing the reset button on the antenna computer.

the observatory clock is off, and the antenna can never reach its intended position and be in the "acquired" status. Backend computer doesn't take data when antenna is not in the "acquired" status.

check if the two clocks on the antenna computer (the two red LED cards) are in sync. If not, reboot the antenna computer and make sure the booted-up system have the clocks in sync.

for some reason the acq_client that UIP runs on alpha1 died. As a result backend computer is not notified of the antenna's acquired status even when the antenna computer is running properly and antenna is driven to the intended position.

reload the antenna computer by "ant/restart/nosync". This will kill and restart all the necessary servers/clients on alpha1.

the hot wheel that UIP commands in and out of the beam during a normal Cal scan is not functioning properly. It's rather rare this happens, but we've seen it.

If you have exhausted the above possible causes, try this one. You'll need to recycle the power on Galil, a box at the top of the rack in the sidecab. It has a green power button. After power on and the LEDs settle down, do a "cal" under UIP and see if it works. If you get an unlikely Tsys, the hot wheel is probably stuck in between in and out poistion. You'll have to recycle the power again until you get the right Tsys with Cal.

"Laser failed..." for a particular AOS

You may get this message while restarting AOSes. This happens when a particular AOS had a sudden current surge that tripped the protection of the AOS.

Go to the AOS room and flip the switch labeled "Manual/Auto" to manual and back to auto on the concerned AOS. Then restart AOSes.

Antenna Computer Related

Antenna isn't tracking

Communication processes between Alpha1 and the antenna computer need refresh

Reload antenna computer under UIP by "ant/nosync". Also read this and this for the latest procedures on restarting antenna computer.

Antenna computer got confused if you killed a process at wrong time. You will know this if you see "From ant:...." when you type any new command after your untimely interruption.

Reload the antenna computer with "ant/nosync". Also read this and this for the latest procedures on restarting antenna computer.

Status display isn't updating

The monitor process on the antenna computer died

Issue UIP command "ant/restart=3/nosync. This will take care of both the console and the sidecab display.

Antenna can't move to your source, as the target position becomes "NaN" or "Inf" in either RA or Dec or both on the status display.

You are caught in one of the rare occasions where the shared data structure used by various programs running on the antenna computer is corrupt. (phew...)

If the antenna is moving, hit the big red STOP button to park the antenna where it is. This will prevent antenna from driving past hard limits in both AZ and ZA. Now antenna computer needs a pretty thorough reload under UIP: "ant/restart=1/nosync". Again refer to this and this for the latest procedures on restarting antenna computer.

ant/restart command hangs on "Restoring display mode"

antenna computer is either crashed or not reachable over the ethernet (try "ping hau" on hapuna)

press the reset button on the antenna computer (labeled hau on the rack in the AOS room). It will take 3-4 minutes for it to come back. Make sure you can ping it, and you can ssh to it as user cso before issuing ant/restart command from UIP. Also make sure the two clocks on the antenna computer (red LEDs) are in sync.

Pointing, etc.

Can't point on a planet

Someone has left chopper running while you're trying to do a five-point pointing in position on/off mode ("five/one" under UIP).

Turn off the chopper by "secondary/stop" under UIP, then proceed to do ""five/one 15".

Sky is rough or the planet is not particularly bright. You'll see some negative numbers at some offset positions while doing five-point pointing.

Try use chopper for pointing instead of doing it in position on/off mode. Start up chopper with "secondary 90 1.123 10 10" and go to the sidecab to tune up the chopper. Then reduce the integration time somewhat (remember a chop-slewy cycle is symmetrical, taking twice as longer than an oo scan cycle), and just "five 15" will do. If you still can't point on a bright planet, such Jupiter or saturn, you should consider doing something else for the night. But if you still want to keep taking data, just look into the pointing log and find an entry that is close to where you want to go on the sky and is for the same receiver. Key in the relevant FAZO/FZAO. Our recent experience with the pointing is that it's very stable and repeatable.

Your pointing offsets are way off compared with those of previous nights

Secondary may not be in lock, being moved from TTL mode (for SHARC) to analog mode. You'll notice this if the on/off LED indicator on the chopper box in sidecab isn't on.

Go to the sidecab, and switch the chopper in proper mode (TTL or analog), then press the on/off button on the chopper box, the LED should go on.

Tertiary mirror may not be properly locked into place.

Make sure the lower end of the tertiary mirror is pushed against the iron bar with the bolt, with which the mirror is screwed in place. Failing this, pointing has been seen off as much as 30".

You may have just experienced one of the numerous volcanic earthquakes on the island.

Inform a CSO staff member of the pointing changes. A new pointing model may be in order.

Shutter Related

Can't open/close shutter. There are red LEDs on on the control panel.

One or more of the stop buttons or interlocks was pressed.

the stop buttons are installed at about a dozen place around the dome. There are also several interlocks on places such as chain hoists and swing doors which, if not properly secured, would interfere with dome/shutter movement. Each of these buttons is indicated by a lit and clearly labeled LED on the control panel in control room. So are some of the interlocks. Find the LEDs that are lit and go to the appropriate location to disengage the stop button or secure the interlocks.

Other shutter malfunction

There could be several shutter drive hydraulics related problems. In the order of descending likelihood:

the hydraulic drive is stuck in low pressure drive. Pump is running, but there is no shutter movement.

Check hydraulic drive pressure at the machine shop control panel and see if the meters read around 600 psi. If so, reset the pump to high pressure drive by depressing the close limit switch at the lower left corner of shutter. This is at the machine shop side of the shutter opening. Look for a cloth hanger like switch.

Pump is not running. The key switch is left in shop control.

On the control panel down in the machine shop, there is a keyed switch which toggles the shutter control between the control room and the machine shop. You would want to moved the keyed switch to control room control.

Pump is not running. Circuit breaker is open.

Reset breaker down in the UPS room. It's on panel P, breaker #4.

Hydraulic drive in over-pressure mode...

Go to machine shop. On the shutter control panel, try press the reset button to reset the over-pressure indicators. you cannot reset the over-pressure indicator, you'd need to power cycle the shutter control panel: in the UPS room, look for circuit breaker panel E (the far left one as you face it); turn circuit breakers #1, #3, and #5 to OFF, then back to ON; back to machine shop, on the shutter control panel if any over-pressure indicators are on, press reset.

If for some reason you still cannot control shutter from control room at this point, try control it manually from the machine shop.

follow this detailed instruction to control shutter manually.

Mechanical problem. Pump is running, but shutter can't be moved.

Look for damages. Check pressures of the hydraulic drive in the machine shop. If they reads 1000 psi or higher, don't attempt to move dome or shutter. Stop observing. Call Allen for his immediate attention.

Dome Related

AZ drive malfunction. Yellow limit LED lit on the control panel.

Dome drive controller tripped.

Reset the dome drive controller by pressing the STOP button on the control panel in the control room or reset the breaker (#1) on panel P down in the UPS room.

Antenna drive controller tripped.

Reset the breaker 7, 9, and 11 at the breaker box outside the Tee Pee (on the back side opposite the Tee Pee door).

Continuing AZ limit problems

Mechanical drag

Do "nolim" under UIP, then try to drive off limit with small moves till tlength (which is displayed on the lower right corner of the status display monitor) is 0.6 or less. Do "lim" under UIP to restore the software limits.

Red Ultimate limit LED lit

Dome or antenna has driven through the software limits, and hit last limit before the hard stops.

This rarely happens. You know you are in serious trouble if it does. Try resetting the dome drive controller and antenna drive controller first (see above). If that doesn't resolve the problem, treating it as a mechanical drag. After issuing "nolim" command under UIP, you'll need to hold the Ultimate limit bypass switch (located below the STOP button) while driving off the limits with small moves (see above). Do not use the hand paddle. Call Allen.

Antenna stopped while tracking or slewing to a source. T_length on the status monitor flashing

One of the dome drive may have tripped. See above two items.

Press then pull out the Red Stop button on the control panel, and see if this resets the red/yellow LED lights on the control panel. Then try drive the antenna (in small steps in AZ) out of the limit. See above two items.

ZA drive malfunction

Antenna drive controller tripped.

Reset breaker 7, 9, and 11 on the breaker box down in the Tee Pee.