Observer Manual

1. Daily Startup Instructions
2. Daily Shutdown Instructions
3. Cryogen Fill Instructions
4. IV Curve Skydip Instructions
5. What to check while observing
6. Revision History

Daily Startup Instructions

1. Check with day crew to see if it is ok to start moving telescope.
2. Pull all stops so that you can move the telescope and open the dome.
3. Open the dome ~1 hr before you would like to start observing; coordinate this with the day crew and make sure you are pointed in a safe direction. You only need to go to ~50% or so. Note in the observing log the time that you open the dome and the fraction.  Make sure there is no sun on the primary or secondary.
4. Log into ALPHA1.submm.caltech.edu as bolocam (at the console or via telnet from another machine). At the prompt, type UIP to start
   UIP. Accept the defaults for the login questions. At the UIP prompt (UIP>) type INSTRUMENT BOLOCAM and immediately check that the antenna computer monitor shows POINTING BOLOCAM and the appropriate focus parameters.  Current values:
   • FOCUS OFFSET = -0.24 mm
   • Y OFFSET = -0.35 mm
5. Check that the fridge cycle has completed or is near completion. You may not move the telescope from the ZA = 55 fridge cycle position until the 3He pump is < 10K.  Note: if pumping on LHe bath replace steps 5 and 6 with Pumping on the LHe Bath.
6. If necessary, do cryogen fills.
   
7. Check that the cap is off and that the mirrors are free of obstructions.
8. Make sure the rotator is at its nominal angle (marks on the rotator). Check that C-clamp is tight.
9. Make sure no cables/hoses have come loose or are binding anywhere.
10. Start slewing over to your first source and finish opening the dome -- this will take a long time, it can be done while the remaining steps are happening. You can use the OBSERVE command to do this, or just type in the az and za commands to get you close. Leave one person dedicated to ensuring no sun problems. Sun should not be allowed to hit the back of the primary or the secondary mirror. Sun is allowed to shine on the back part of the secondary structure (the white part), though be careful because the sun may move DOWN the structure as you observe if the source is rising too quickly.  In step 15, you will start the webcam utility, which will let you monitor the sun exposure from the control room.  However, note that the webcams do not let you see the left (toward decreasing azimuth) side of the secondary structure!
    
11. DAS computer prep:
    • Create directory D:\das_data\YYYYMMDD where YYYYMMDD is the UT day of the data you are about to take.
    • Delete enough old data that there is about 3 GB or more free.
      
    • Go to the Labview DAS program, BCAM_DAS_20011206.vi (on the PC's desktop), and change the write directory to the above one you just created.
      
    • Make sure Trigger switch is UP.
      
    • Do quick check of front of DAS for loose cables. 
    • Do not start the DAS yet!
12. Start the various programs on allegro (rotator controller, rotator log writer, das file copier, pointing file copier):
    • Log in to kilauea (the computer with the dual screens) as bolocam, password is the usual summit password.
    • Start a terminal window in the right screen (the DISPLAY variable for this screen is :0.1).  From this window, SSH to allegro as observer (ask someone for the password, it's not the same as for the other summit machines).  As of 12/2003, the default for SSH is not to do X forward, so use ssh -X to ensure X forwarding.
      
    • At the allegro shell prompt, type start_tel_util YYYYMMDD (YYYYMMDD is UT, see the antenna computer display).  Four windows should pop up: rotator, write_log, dirsync.py,  and header_copy.py.  In the rotator window, hit s so that the rotator will not rotate. Wait until you see some text in all the windows to make sure that nothing is going wrong.
13. Go back upstairs and start DAS. Wait until you see data. All the DC lockin channels should be ~few volts and bit-limited.  These are channels, 32-55 (hex 1), 96-119 (hex 2) and so on.  The bias monitor DC lockin channels are also good to look at because they won't drift due to base temperature or loading changes.  They are channels 377-382.  These are especially good for checking whether the flaky bias problem has recurred.  Turn off the monitor, all lights inside dome, and go back to control room.
14. Start QuickLook:
    
    1. In the SSH client you started up in step 12, type run_gb &
    2. From the Settings menu tab, select Preferences and select the appropriate source file directories:
       • DAS directory: /smb/andante/YYYYMMDD
       • Telescope pointing log directory: /data/plog
       • rotator log directory: /data/encdir/YYYYMMMDD
       • configuration directory: leave default value
    3. Go to the File menu tab and hit Go. Make sure you see something reasonable (see below).
       
    4. QuickLook occasionally dies. If you notice it is not updating, quit it and restart as follows:
    • Go to File menu tab and hit Exit
    • Repeat step 14 to restart.
15. If you are observing during daylight hours, you will want to use the webcam utility.  It shows the webcams that view the secondary and the back of the primary.  It updates more quickly than the webcam display on the CSO web page.  To start it, in the SSH client you started in step 12, type webcam &,
    
16. Go to UIP and start your observing.
17. Start merging, slicing, and automated analysis
    1. Start a new terminal window on kilauea in the second desktop of the right display.  Don't forget to use the second desktop -- otherwise you will suffer from window chaos. 
    2. Type start_merge YYYYMMDD.  A window labelled merge should appear and you should see it start updating as it merges the pointing, rotator, and DAS files.
       
    3. Type start_autos.  Eight (yes, eight!) windows should appear.  In each window, IDL will start up and a program will start running.  These programs are:
    • slicing of merged files into single-observation files:  run_auto_slice_files
      
    • cleaning of pointing files: run_auto_clean_files_ptg
    • mapping of pointing files: run_auto_map_files_ptg
    • centroiding of pointing maps: run_auto_centroid_files
    • cleaning of science field files: run_auto_clean_files_blankfields
      
    • mapping of science field files: run_auto_map_files_blankfields
    • diagnostics on science field cleaned files: run_auto_diag_clean_blankfields
    • diagnostics on science field map files: run_auto_diag_map_blankfields
    • For instructions on monitoring these routines, see the section below.
18. Check the status of the previous night's backups and do the second backup if necessary.  See the TapeBackup page for instructions.
    

Daily Shutdown Instructions

1. Go upstairs and stop DAS.
2. Start closing dome and tip back to ZA = 5 for cryo fills. Be careful of sun during tip back and dome close.
3. Fill LHe and LN (cryogen fills). You should push one of the red STOP buttons while doing a cryo fill. This of course means that you have to wait until the dome is sufficiently closed that there's no sun danger.  Note: if pumping on LHe bath replace step 3 with Pumping on the LHe Bath.
4. Make sure the bath caps are on and the appropriate vent lines attached and that the vent lines are unobstructed and have good gas outflow. Pull stop button, tip to ZA = 55 for fridge cycle, and type STOW at the UIP prompt to move the telescope to the stow position in azimuth.
   
5. Restart dome closing if not already closed.
6. Make sure fridge web page is visible on web. If it has problems, reboot the fridge computer. To restart fridge cycle program, click on the fridge_cycle_all_therms icon on the desktop and select "Jason Glenn" as the user.  (Note, if you are using the alternate  no IC 3He cycle version, you should click on fridge_cycle_all_therms_no_IC3He.)
   
7. Start fridge cycle, possibly with start delay -- you want the fridge to be cold when 1st shift arrives, and cycle takes 4 hours including stabilization at end. (If you are pumping, you can delay it more -- the 1st shift can start pumping before the fridge has fully equilibrated.  Also, the cycle takes less time if no IC 3He cycle is done.)  Remember to press the FRIDGE IS ON/OFF button and provide a data filename of the form cycle_YYYYMMDD.txt BEFORE starting the program.  The program ignores these inputs once it has read them on startup.  Start the program by clicking on the arrow in the upper left corner. Advise day crew to leave telescope at ZA = 55 for the first 3 hrs of cycle.
8. Kill the 4 windows with the names rotator, write_log, dirsync.py, and header_copy.  It is especially important that you kill header_copy to prevent the pointing files from being overwritten by the next day's files.
9. If the merging is caught up to the end of the day, kill the merge window.
10. If the slicing is caught up to the end of the day (all but the last observation sliced), then hit c in the IDL session running the slicing to have it slice the last file.  Quit IDL and close the window once the last file is sliced.
11. Once the other analysis routines are caught up, hit q in each of the windows to stop them.  Exit all the IDL sessions and close the windows.  If problems resulting in the analysis being far from done, you should quit the sessions in the same way regardless and restart all these routines from the office at HP (you can just start them up as you do at the start of the day (see above) -- they will figure out which files have not yet been analyzed).
    
12. Start a backup of the raw data. Insert the tape in the tape drive. At any shell prompt on allegro, type
         backup_day YYYYMMDD
where YYYYMMDD is the UT day of the data just taken. If you have enough time, you can run the second copy also. There should be two tapes, labeled copy 1 and copy 2. To rewind and eject the first tape, an any shell prompt type:
     mt -f /dev/nst0 rewind
     mt -f /dev/nst0 offline
Then insert the second tape and run the same command as above. Advise Sunil or Alexey of any problems.

13. Make sure the dome is closed, the telescope at ZA = 55, and the dewar cryogen baths are capped and have unobstructed vent lines with outflowing gas.
14. Exit UIP, log out of any open consoles, push a stop button, turn off all the lights, lock the door, and look forward to a healthy and delicious breakfast at HP followed by a restful sleep in their fine accommodations.
    

Cryogen Fill Instructions

• Tip to ZA = 5
• Push a stop
• Defrost baths that need fills.  When defrosting, be sure to heat the metal, not the rubber hoses or the RTV at the bottom of the inlets!  A loss of vacuum could really ruin your day.
  
• REMEMBER, LN BATH IS THE CENTER BATH!
• Fill LHe from dewar on alidade and using BOLOCAM transfer tube (wide bore). Fill LN from 3rd floor with long fill line. Do fills simultaneously to save time.
• Defrost bath inlets (heat the metal of the inlet) and replace bath caps.
• Check vent lines are free of obstructions and gas is flowing out.
• Pull stop afterward.

IV Curve Skydip Instructions

1. Slew to whatever azimuth you want to do the skydip at.  Presumably you are doing it in conjunction with observations of some sort of calibrator source, so you should do it at the same azimuth.
   
2. Execute the macro by issuing the command
   
   UIP> EXEC BOLOCAM_SKYDIP_LOADCURVE.MAC
   
3. Wait while the telescope slews to ZA = 5 degrees
   
4. Go to the alidade platform, open the back of the electronics box, and flip the following switches on the bias board (the board that is 2nd from the left, it has lots of switches and knobs).  Note that there is a diagram of the front panel on the outside of the electronics box for reference.  You have 2 minutes to do this, don't dilly-dally!
   
• AC/DC switch (TOP switch, right underneath the 9-pin D connector output) should flip to DC mode (DOWN position)
  
• EXT/INT switch (2 switches below the AC/DC switch) to EXT (UP position)
• Leave the e-box open
5. Go back inside, and wait for the FLSIGNAL 128 /SET command to appear on the screen followed by the line
   
   C OBSERVER SHOULD NOW GO OUTSIDE AND FLIP AC/DC
   
6. When the above lines appear, go back outside and flip the AC/DC switch to AC (UP position) and close the e-box.  Again, you have 2 minutes to do this.
7. Wait until the macros runs through all the elevations and the FLSIGNAL 128 /RESET command to appears on the screen, followed by the comment line
   
   C OBSERVER SHOULD GO OUTSIDE AND SET EXT/INT
   
   and the bell indicating the macro is finished.
   
8. When the macro finishes, go back outside, open the e-box and flip the EXT/INT switch to INT mode (DOWN position).  Close the e-box.
9. Continue with observing as before.
   

• Make sure that the antenna computer is tracking the intended source, and that ZA < 60.
  
• Check that write_log, dirsync.py, and header_copy.py are updating.  These are very important: if they are not updating, then we may be losing data (something may have crashed).
  
• Watch to see if the plots in QuickLook look as they should:
• AC broadband noise ~.02 Volts (i.e., above and beyond common-mode sky noise)
  
• DC level of a few Volts
• RGRT = - (resistance/200kohms) x 10 Volts (you can read the resistance bridge directly -- it is mounted in the rack with the lockins).
  
• observation number updating
• reasonable values for tau, temperature, humidity
• Make sure that the merging, slicing, cleaning, mapping, centroiding, and diagnostic plotting windows are all updating.  The cleaning or mapping may stop if a source is encountered that is not in their params files, which will result in an error message indicating this.  See the Troubleshooting section for instructions on dealing with this. 
  
• Keep an eye on the diagnostic outputs.  There are two kinds:
• Pointing sources: in /data10/200310/centroid/, there are directories for each source.  Those directories will have .ctr files that provide centroid information for each bolometer for the pointing observations.  Make sure new files are appearing for the sources you have observed.
• Science fields: Diagnostic plots for each observation are saved to /data10/200310/psd_plot/ and /data10/200310/map_sum_plot/, with subdirectories for each source.  For each observation, there will be 10 files in the psd_plot directory and 2 files in the map_sum_plot directory, listed below.  Check that these files are behaving reasonably.  Everything will degrade if the weather gets worse, but make sure things do not change too much (< factor of 2 in sensitivity and PSDs).
  
• psd_plot/object_name/YYMMDD_OOO_clean1_psd_bias.eps: PSDs of the bias monitors
  
• psd_plot/object_name/YYMMDD_OOO_clean1_psd_hex_H.eps (6 of them H = a, b, c, d, e, f): PSDs of the bolometers, with point source profile (green) and optimally filtered PSD (red) overlaid
  
• psd_plot/object_name/YYMMDD_OOO_clean1_sens.eps: Optimally filtered point-source sensitivity based on above PSDs
• psd_plot/object_name/YYMMDD_OOO_clean1_pca_evals.eps: Scan-by-scan histograms of the eigenvalues from PCA cleaning, with the eigenvalue cut displayed (we discard PCA eigenvectors with eigenvalues above the cut).
  
• psd_plot/object_name/YYMMDD_OOO_clean1_pca_sum.eps: Summary histograms of the number of PCA eigenvectors that are cut, the value of the cut on the eigenvalue distribution, median of the eigenvalue distribution, and the width of the eigenvalue distribution.  The histograms have one entry for each scan in the observation.
• map_sum_plot/object_name/YYMMDD_OOO_clean1_map_diag_hist.eps: Histograms of the maps for the observation:
• histogram of the coverage map (all pixels in naive and optimally filtered maps, good coverage region of naive and optimally filtered maps).  Should have a nice peak.
• cumulative distribution for the coverage histograms
• histograms of pixel values (again, full naive map, full optimally filtered map, good coverage region of naive map, good coverage region of optimally filtered map)
• histograms of sensitivity ( = pixel value x √time in pixel ) (all four versions)
• map_sum_plot/object_name/YYMMDD_OOO_clean1_map_diag_3d.eps: Maps:
• naive map (square indicates good coverage region)
  
• naive coverage map (square indicates good coverage region)
  
• PSD of naive map (good coverage region) (greyscale, surface, and contour plots)
• optimal filter in freequency space
  
• optimally filtered map
• optimally filtered coverage map
• PSD of optimally filtered map
  
• In the afternoon and morning - be sure the sun is not hitting the primary or secondary.
• If pumping on the LHe bath, watch for ice plugs; see Pumping on the LHe Bath.
  

• 2003/11/02 JS/SG
  First web version
  
  
• 2003/11/04 SG
  Add Troubleshooting section.
  
  
• 2003/12/07 SG
  Move Troubleshooting section to a separate page.
  
  
• 2003/12/08 SG
  Move Pumping on the LHe Bath section to a separate page.  Move tape backup instructions to a separate page
  
  
• 2004/01/31 SG
  Add link back to main page, add instructions for IV curve skydip.
  
  
• 2004/02/03 SG
  Change instructions to assume analysis is run on kilauea instead of champinux