Task 1729422515

Name: h1_0184.80_O3aLC01Cl1In0__O3ASBu_185.00Hz_48275_0
Workunit ID: 873719234
Created: 13 Feb 2025 23:16:31 UTC
Sent: 13 Feb 2025 23:16:32 UTC
Report deadline: 20 Feb 2025 23:16:32 UTC
Received: 15 Feb 2025 9:38:06 UTC
Server state: Over
Outcome: Success
Client state: Done
Exit status: 0 (0x00000000)
Computer: 13063321
Run time (sec): 4,417.86
CPU time (sec): 4,386.82
Peak working set size (MB): 1226.15
Peak swap size (MB): 44494.78
Peak disk usage (MB): 30.46
Validation state: Valid
Granted credit: 20,000
Application: All-Sky Gravitational Wave search on O3 v1.16 (GW-cuda-3)
x86_64-pc-linux-gnu
Stderr output

<core_client_version>7.20.5</core_client_version>
<![CDATA[
<stderr_txt>
putenv 'LAL_DEBUG_LEVEL=3'
2025-02-14 21:38:37.5443 (49553) [normal]: This program is published under the GNU General Public License, version 2
2025-02-14 21:38:37.5444 (49553) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2025-02-14 21:38:37.5444 (49553) [normal]: This Einstein@home App was built at: Mar  7 2024 14:38:26

2025-02-14 21:38:37.5444 (49553) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-cuda-3'.
2025-02-14 21:38:37.5452 (49553) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, CUDA, X64, SSE, SSE2, GNUC X86 GNUX86
2025-02-14 21:38:37.5452 (49553) [debug]: glibc version/release: 2.36/stable
2025-02-14 21:38:37.545231 - mytime()
2025-02-14 21:38:37.5454 (49553) [debug]: Set up communication with graphics process.
2025-02-14 21:38:37.5578 (49553) [normal]: Parsed user input successfully
Code-version: %% LAL: 7.1.4.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALPulsar: 3.1.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALApps: 7.3.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)

2025-02-14 21:38:37.5579 (49553) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 10000.
2025-02-14 21:38:37.5582 (49553) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0184.80_O3aLC01Cl1In0__O3ASBu_185.00Hz_48275_0_0.tmp' for writing ...
2025-02-14 21:38:37.7838 (49553) [normal]: Reading input data ... 
2025-02-14 21:38:37.7838 (49553) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0184.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0184.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0185.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0185.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0185.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0185.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0185.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.20_O3aLC01Cl1In0' into catalog ...2025-02-14 21:38:37.9479 (49553) [normal]: done.
2025-02-14 21:38:37.9479 (49553) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
2025-02-14 21:38:40.4316 (49553) [normal]: Search FstatMethod used: 'ResampGPU'
2025-02-14 21:38:40.4316 (49553) [normal]: Recalc FstatMethod used: 'DemodGPU'
2025-02-14 21:38:40.4316 (49553) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'NVIDIA GeForce GTX 980'
2025-02-14 21:38:40.4316 (49553) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'CUDA'
2025-02-14 21:38:40.4317 (49553) [normal]: GPU version is used for the semi-coherent step!
2025-02-14 21:38:56.3355 (49553) [normal]: Number of segments: 18, total number of SFTs in segments: 5748
2025-02-14 21:38:56.3395 (49553) [normal]: Finished reading input data.
2025-02-14 21:38:56.3395 (49553) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
2025-02-14 21:38:56.3396 (49553) [normal]: dFreqStack = 1.000000e-06, df1dot = 7.500000e-11, df2dot = 0.000000e+00, df3dot = 0.000000e+00
2025-02-14 21:38:56.3396 (49553) [normal]: Setup, N = 18, T = 878400 s, Tobs = 15809012 s, gammaRefine = 250, gamma2Refine = 1098, gamma3Refine = 1

DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
2025-02-14 21:39:09.1047 (49553) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
2025-02-14 21:39:09.1533 (49553) [normal]: Cpt:0,  total:9000,  sky:1/225,  f1dot:1/40

0.2025-02-14 21:39:09.1539 (49553) [normal]: CG:18010710 FG:1000000  f1dotmin_fg:-2.73735059761e-09 df1dot_fg:2.988047808765e-13 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
.......................................
1.........c
...............................
2........................................
3........................................c

4........................................
5........................................
6..............................c
..........
7........................................
8........................................
9......................c
..................
10........................................
11........................................
12..............c
..........................
13........................................
14........................................
15......c
..................................
16........................................
17.......................................c
.
18........................................
19........................................
20...............................c
.........
21........................................
22........................................
23........................c
................
24........................................
25........................................
26................c
........................
27........................................
28........................................
29.........c
...............................
30........................................
31........................................
32..c
......................................
33........................................
34.....................................c
...
35........................................
36........................................
37..............................c
..........
38........................................
39........................................
40.........................c
...............
41........................................
42........................................
43..................c
......................
44........................................
45........................................
46............c
............................
47........................................
48........................................
49......c
..................................
50........................................
51........................................
52.c
.......................................
53........................................
54.....................................c
...
55........................................
56........................................
57...............................c
.........
58........................................
59........................................
60...........................c
.............
61........................................
62........................................
63.......................c
.................
64........................................
65........................................
66..................c
......................
67........................................
68........................................
69.............c
...........................
70........................................
71........................................
72.......c
.................................
73........................................
74........................................
75.c
.......................................
76........................................
77....................................c
....
78........................................
79........................................
80...............................c
.........
81........................................
82........................................
83.........................c
...............
84........................................
85........................................
86...................c
.....................
87........................................
88........................................
89...............c
.........................
90........................................
91........................................
92............c
............................
93........................................
94........................................
95........c
................................
96........................................
97........................................
98..c
......................................
99........................................
100....................................c
....
101........................................
102........................................
103..............................c
..........
104........................................
105........................................
106........................c
................
107........................................
108........................................
109.....................c
...................
110........................................
111........................................
112...............c
.........................
113........................................
114........................................
115..........c
..............................
116........................................
117........................................
118...c
.....................................
119........................................
120.......................................c
.
121........................................
122........................................
123....................................c
....
124........................................
125........................................
126................................c
........
127........................................
128........................................
129...........................c
.............
130........................................
131........................................
132......................c
..................
133........................................
134........................................
135..................c
......................
136........................................
137........................................
138..............c
..........................
139........................................
140........................................
141...........c
.............................
142........................................
143........................................
144........c
................................
145........................................
146........................................
147.....c
...................................
148........................................
149........................................
150.c
.......................................
151........................................
152.....................................c
...
153........................................
154........................................
155................................c
........
156........................................
157........................................
158............................c
............
159........................................
160........................................
161.......................c
.................
162........................................
163........................................
164..................c
......................
165........................................
166........................................
167.............c
...........................
168.....................
2025-02-14 22:38:22.686270
-- signal handler called: signal 15
XLAL Error - XLALGetVectorFromREAL4VectorGPU (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/GPUUtils.c:363): GPU is not initialized!
XLAL Error - XLALGetVectorFromREAL4VectorGPU (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/GPUUtils.c:363): Generic failure
XLAL Error - UpdateSemiCohToplistsOptimTriple_GPU (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:4401): Check failed: XLALGetVectorFromREAL4VectorGPU ( &in->sumTwoF[0], in->sumTwoF_GPU, numberOfCandidates) == XLAL_SUCCESS
XLAL Error - UpdateSemiCohToplistsOptimTriple_GPU (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:4401): Internal function call failed: Generic failure
XLAL Error - MAIN (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:2326): Check failed: UpdateSemiCohToplistsOptimTriple_GPU ( compartment, compartment2, compartment3, &finegrid, f1dot_fg, f2dot_fg, f3dot_fg, usefulParams.NSegmentsInvX, XLALUserVarWasSet(&uvar->f2dot), XLALUserVarWasSet(&uvar->f3dot), uvar->computeBSGL, uvar->getMaxFperSeg, uvar->writeLeanerOutput ) == XLAL_SUCCESS
XLAL Error - MAIN (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:2326): Internal function call failed: Generic failure
2025-02-14 22:38:22.6864 (49553) [CRITICAL]: ERROR: MAIN() returned with error '1'
Code-version: %% LAL: 7.1.4.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALPulsar: 3.1.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALApps: 7.3.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)

FPU status flags: 
2025-02-14 22:38:22.6868 (49553) [debug]: worker done. return(1) to caller
2025-02-14 22:38:22.6868 (49553) [normal]: done. calling boinc_finish(1).
22:38:22 (49553): called boinc_finish
XLALCUDARelease: Total malloc from <TYPE>VectorGPU is greater than 0: 2077284925XLALCUDARelease: Total malloc from COMPLEX8FFTPlanGPU is greater than 0: 671088649 stack frames obtained for this thread:
Use gdb command: 'info line *0xADDRESS' to print corresponding line numbers.
../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-cuda-3[0x4301f7]
/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f9927a23050]
/lib/x86_64-linux-gnu/libc.so.6(clock_nanosleep+0x65)[0x7f9927ab6505]
/lib/x86_64-linux-gnu/libc.so.6(nanosleep+0x13)[0x7f9927abae13]
/lib/x86_64-linux-gnu/libc.so.6(usleep+0x45)[0x7f9927ae6205]
../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-cuda-3[0x66942d]
../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-cuda-3[0x657a2d]
/lib/x86_64-linux-gnu/libc.so.6(+0x891c4)[0x7f9927a701c4]
/lib/x86_64-linux-gnu/libc.so.6(+0x10985c)[0x7f9927af085c]

End of stacktrace
Stack trace of LAL functions in worker thread:
InitDopplerSkyScan at /home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalpulsar/lib/DopplerScan.c:291
At lowest level status code = 0: NO LAL ERROR REGISTERED
putenv 'LAL_DEBUG_LEVEL=3'
2025-02-15 03:35:12.6925 (1217) [normal]: This program is published under the GNU General Public License, version 2
2025-02-15 03:35:12.6925 (1217) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2025-02-15 03:35:12.6925 (1217) [normal]: This Einstein@home App was built at: Mar  7 2024 14:38:26

2025-02-15 03:35:12.6925 (1217) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-cuda-3'.
2025-02-15 03:35:12.6964 (1217) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, CUDA, X64, SSE, SSE2, GNUC X86 GNUX86
2025-02-15 03:35:12.6965 (1217) [debug]: glibc version/release: 2.36/stable
2025-02-15 03:35:12.696510 - mytime()
2025-02-15 03:35:12.6967 (1217) [debug]: Set up communication with graphics process.
2025-02-15 03:35:12.7421 (1217) [normal]: Parsed user input successfully
Code-version: %% LAL: 7.1.4.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALPulsar: 3.1.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALApps: 7.3.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)

2025-02-15 03:35:12.7422 (1217) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 10000.
2025-02-15 03:35:12.7424 (1217) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0184.80_O3aLC01Cl1In0__O3ASBu_185.00Hz_48275_0_0.tmp' for writing ...
2025-02-15 03:35:12.9771 (1217) [normal]: Reading input data ... 
2025-02-15 03:35:12.9771 (1217) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0184.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0184.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0185.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0185.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0185.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0185.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0185.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.20_O3aLC01Cl1In0' into catalog ...2025-02-15 03:35:13.2010 (1217) [normal]: done.
2025-02-15 03:35:13.2011 (1217) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
2025-02-15 03:35:15.8098 (1217) [normal]: Search FstatMethod used: 'ResampGPU'
2025-02-15 03:35:15.8099 (1217) [normal]: Recalc FstatMethod used: 'DemodGPU'
2025-02-15 03:35:15.8099 (1217) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'NVIDIA GeForce GTX 1660 SUPER'
2025-02-15 03:35:15.8099 (1217) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'CUDA'
2025-02-15 03:35:15.8099 (1217) [normal]: GPU version is used for the semi-coherent step!
2025-02-15 03:35:31.5530 (1217) [normal]: Number of segments: 18, total number of SFTs in segments: 5748
2025-02-15 03:35:31.5575 (1217) [normal]: Finished reading input data.
2025-02-15 03:35:31.5575 (1217) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
2025-02-15 03:35:31.5576 (1217) [normal]: dFreqStack = 1.000000e-06, df1dot = 7.500000e-11, df2dot = 0.000000e+00, df3dot = 0.000000e+00
2025-02-15 03:35:31.5576 (1217) [normal]: Setup, N = 18, T = 878400 s, Tobs = 15809012 s, gammaRefine = 250, gamma2Refine = 1098, gamma3Refine = 1

DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
2025-02-15 03:35:44.1553 (1217) [normal]: Successfully read checkpoint:6693
2025-02-15 03:35:44.2288 (1217) [normal]: Cpt:6693,  total:9000,  sky:168/225,  f1dot:14/40

167.2025-02-15 03:35:44.2292 (1217) [normal]: CG:18010710 FG:1000000  f1dotmin_fg:-1.76235059761e-09 df1dot_fg:2.988047808765e-13 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
..........................
168........................................
169................c
........................
170........................................
171........................................
172........................................
173...............................c
.........
174........................................
175........................................
176........................................
177........................................
178......c
..................................
179........................................
180........................................
181........................................
182......................c
..................
183........................................
184........................................
185........................................
186......................................c
..
187........................................
188........................................
189........................................
190........................................
191..............c
..........................
192........................................
193........................................
194........................................
195..............................c
..........
196........................................
197........................................
198........................................
199........................................
200......c
..................................
201........................................
202........................................
203........................................
204......................c
..................
205........................................
206........................................
207........................................
208......................................c
..
209........................................
210........................................
211........................................
212........................................
213..............c
..........................
214........................................
215........................................
216........................................
217...............................c
.........
218........................................
219........................................
220........................................
221........................................
222......c
..................................
223........................................
224........................................
2025-02-15 03:48:53.0854 (1217) [normal]: Finished main analysis.
2025-02-15 03:48:53.0854 (1217) [normal]: Recalculating statistics for the final toplist(s)...
2025-02-15 03:48:56.9455 (1217) [normal]: Recalculation step: 20000/200000
2025-02-15 03:49:00.7782 (1217) [normal]: Recalculation step: 40000/200000
2025-02-15 03:49:04.6331 (1217) [normal]: Recalculation step: 60000/200000
2025-02-15 03:49:08.4891 (1217) [normal]: Recalculation step: 80000/200000
2025-02-15 03:49:12.3497 (1217) [normal]: Recalculation step: 100000/200000
2025-02-15 03:49:16.2067 (1217) [normal]: Recalculation step: 120000/200000
2025-02-15 03:49:20.0572 (1217) [normal]: Recalculation step: 140000/200000
2025-02-15 03:49:23.9086 (1217) [normal]: Recalculation step: 160000/200000
2025-02-15 03:49:27.7502 (1217) [normal]: Recalculation step: 180000/200000
2025-02-15 03:49:31.5870 (1217) [normal]: Recalculation step: 200000/200000
2025-02-15 03:49:31.7844 (1217) [normal]: Finished recalculating toplist statistics.
2025-02-15 03:49:31.7844 (1217) [normal]: Finished in 859.04 s with peak RAM usage: 1455.1 MB on CPU 'AMD Ryzen 7 2700X Eight-Core Processor         ', peak VRAM usage: 2045.1 MB on GPU Device: 'NVIDIA GeForce GTX 1660 SUPER' with backend: 'CUDA'.
2025-02-15 03:49:31.8179 (1217) [debug]: Writing output ... Closing temp output file '../../projects/einstein.phys.uwm.edu/h1_0184.80_O3aLC01Cl1In0__O3ASBu_185.00Hz_48275_0_0.tmp' ... renaming temp output file '../../projects/einstein.phys.uwm.edu/h1_0184.80_O3aLC01Cl1In0__O3ASBu_185.00Hz_48275_0_0.tmp' to '../../projects/einstein.phys.uwm.edu/h1_0184.80_O3aLC01Cl1In0__O3ASBu_185.00Hz_48275_0_0' ... done.
2025-02-15 03:49:32.0433 (1217) [normal]: Restarted from checkpoint 6693

DEPRECATION WARNING: program has invoked obsolete function FreeDopplerSkyScan(). Please see XLALDestroyDopplerSkyScan() for information about a replacement.
Code-version: %% LAL: 7.1.4.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALPulsar: 3.1.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALApps: 7.3.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)

FPU status flags: 
2025-02-15 03:49:32.1305 (1217) [debug]: worker done. return(0) to caller
2025-02-15 03:49:32.1305 (1217) [normal]: done. calling boinc_finish(0).
03:49:32 (1217): called boinc_finish

</stderr_txt>
]]>