Posts

Posted on

156 GiB on 62 GiB of VRAM: Making DeepSeek-V4-Flash Practical on a Four-GPU Ampere Workstation

The FreeToken logo, white and blue wordmark on a dark background, with the tagline Bring Frontier to Edge

The workstation under my desk is a dual-socket Lenovo with four PCIe-attached RTX A4000 cards. Sixteen gigabytes of VRAM each, 62.40 GiB in aggregate, Ampere, no native FP4 tensor cores. The DeepSeek-V4-Flash-0731 checkpoint on its local disk is 156 GiB. The official model card demonstrates DSpark decoding for this model on a four-GPU GB300 node and recommends specialized MXFP4 backends. I have neither.

An NVIDIA RTX A4000 single-slot blower-style graphics card
An NVIDIA RTX A4000: a single-slot blower-style Ampere board with 16 GiB of GDDR6 and 448 GB/s of memory bandwidth. Four of them sit in PCIe slots in this workstation with no NVLink between them, so the 62.40 GiB total is spread across four independent memory spaces rather than one pool. Ampere has no hardware FP4 support, so every FP4 expert weight in this checkpoint is decoded in software before it reaches the math.

The first launch failed before CUDA graph capture. A single-rank FreeToken process asked for a minimum cache plan of 9.14 GB from a remaining device budget of 4.09 GB, and stopped. No launch flag fixes that, so feasibility was a systems problem from the start.

Six weeks later the same machine serves the unmodified checkpoint at roughly 28–30 tok/s on structured code generation, against an approximately 18 tok/s target-only baseline. The work went upstream as three stacked pull requests. The full manuscript, with every measurement, threat to validity, and artifact hash, is available as a PDF. This post is the shorter version, built around the paper’s main architecture figure.

FreeToken 0.1.2 startup banner showing model DeepSeek-V4-Flash-0731, TP=4, four RTX A4000 cards totaling 62.4 GiB, 40 physical cores, 502 GiB RAM, two NUMA nodes with rank-to-node mapping 0 0 1 1, followed by log lines showing hybrid MoE backend selection and per-rank NUMA first-touch-local bank placement
The startup banner reports the configuration in full: DeepSeek-V4-Flash-0731 at TP=4, four RTX A4000 cards for 62.4 GiB of VRAM, 40 physical cores and 502 GiB of RAM across two NUMA nodes, and the rank-to-node map 0 0 1 1 that Figure 1 draws. Below it the engine selects its own backends: benchbw profile recommends hybrid for 'ds_fp4' experts on this GPU, then Auto-selected MoE backend: hybrid and the sparse DSV4 attention path. The four NUMA rankN/4 lines report the result of the binding order. Each rank names its node, its 20-core CPU range, and banks first-touch local, meaning its expert pages were written for the first time by a thread that was already pinned to the right socket. Click any log screenshot in this post to read it full size.

What FreeToken already does

None of this starts from zero. FreeToken is an edge-native MoE serving system from Yang, Fan, Pan and colleagues at Berkeley and UT Austin, published as arXiv:2608.16157, and its central idea is that a personal machine should be treated as a unified elastic platform rather than as a small GPU. Their overview figure is the design my work sits on top of:

Figure 2 of the FreeToken paper, the FreeToken overview. Panel 1, prefill, shows full-layer double buffering of PCIe expert loads against GPU compute, and recurrent-state checkpoints anchored at special-token boundaries so an edited context re-prefills only the new suffix. Panel 2, decode, shows a router sending token t to 12 experts of which 8 hit the GPU LRU expert cache, and the 4 misses being divided by q star equals m times B_PCIe over B_Host into one PCIe cache fill and three in-place CPU expert computations, whose partial outputs merge exactly into the layer output
Figure 2 of the FreeToken paper, reproduced with its original caption. The decode panel is the part my work builds on. The four routed-expert misses are divided by q* = m * B_PCIe / B_Host between one cache fill over PCIe and three experts executed in place on the CPU, using bandwidths profiled on the deployed machine, and the two partial outputs merge exactly. That is the same split rule that lands at 28% on my workstation.

Figure and caption reproduced from: S. Yang, X. Fan, M. Pan, H. Xi, Z. Wang, S. Sun, K. Keutzer, S. Han, M. Zaharia, C. Xu, and I. Stoica, “FreeToken: Efficient edge-native MoE serving with bandwidth-adaptive execution,” arXiv preprint arXiv:2608.16157, 2026. [Online]. Available: https://arxiv.org/abs/2608.16157

What that upstream design did not have was a DeepSeek-V4-Flash path that works across four Ampere GPUs with exact DSpark decoding. That is what the three pull requests add, and the rest of this post is the shape of that work.

The architecture, in one diagram

Everything below follows from this figure. It is Figure 1 of my paper, and it shows where each token’s work physically happens on this machine.

Architecture diagram: two NUMA nodes each holding 73.31 GiB of expert banks feed four RTX A4000 GPUs by CPU compute and 28 percent PCIe fetch; the four GPUs feed a TP completion in-place all-reduce, which feeds the DSpark controller doing adaptive width selection or target-only fallback
Figure 1 of the paper: NUMA-local expert ownership, hybrid CPU–GPU miss service, and TP completion. The two green boxes are the NUMA nodes, each holding half the routed-expert population in host memory. Each node feeds its two local GPUs along two arrows with different meanings. CPU compute means the expert is executed in place on host cores; 28% fetch means its bytes are pulled across PCIe and executed on the GPU. That is the FreeToken miss split from the figure above, now measured on this hardware and replicated per socket. The four blue GPU boxes converge on a single in-place all-reduce that completes the tensor-parallel layer, and the gold DSpark controller below it decides how much speculative work the next step is worth. The diagram shows control and data dependence, not physical bus topology.

The path runs from top to bottom. Each NUMA node owns 73.31 GiB of routed-expert weights in host memory, first-touched after its ranks are bound to that socket. Ranks 0 and 1 live on node 0 with GPUs 0 and 1; ranks 2 and 3 live on node 1 with GPUs 2 and 3. When a token routes to an expert that is not resident in GPU cache, that miss is split: about 28% of the missing bytes are fetched over PCIe and executed on the GPU, and the rest are executed in place on the CPU by AVX-512 workers reading the packed MXFP4 banks directly. Both paths launch concurrently. The four rank-local partial results are completed by one in-place all-reduce, and the DSpark controller sits on top deciding, per step, how many speculative proposals are worth verifying.

The rest of this post covers why each of those six mechanisms is present.

Why it would not run

The capacity inequality settles the question on its own:

M_{\text{checkpoint}} = 156\ \text{GiB} \;\gt\; M_{\text{VRAM,total}} = 62.40\ \text{GiB}, \qquad \text{ratio} = 2.5

Aggregate VRAM is only useful once a runtime assigns disjoint ownership to the four devices. A single-rank process sees one 16 GiB card, not 62.40 GiB of pooled memory, and setting a tensor-parallel size was not enough by itself, because the then-current DeepSeek-V4 model shapes and loading path assumed one rank and could replicate or retain the largest tensors.

One detail is invisible until you measure it. Slicing a quantized weight with a contiguous narrow() can still return a view, and holding the view holds the entire parent allocation alive. Keeping views instead of cloning into independent storage cost 5.1 GiB per GPU at TP=4. On a 16 GiB device that single line is a third of the budget.

The other rule the loader enforces is that quantized partitioning must preserve blocks. A requested local dimension that cuts an FP4 32-value block, or an FP8 128-value scale grid, is rejected before the model is constructed rather than silently producing garbage. Weights and their scale grids are sliced on the same axis.

MXFP4 is not NVFP4, and the difference matters on Ampere

The routed experts in this checkpoint are packed E2M1 values with one E8M0 scale shared by every block of 32 values. Let c_{b,i} be the four-bit code at position i of block b, let e_b be the unsigned E8M0 exponent code, and let v(c) be the E2M1 lookup value. The decoded weight is

\widehat{w}_{b,i} \;=\; v(c_{b,i})\,2^{e_b-127}, \qquad i = 0,\dots,31

with the E2M1 magnitude set \{0, 0.5, 1, 1.5, 2, 3, 4, 6\} plus one sign bit. Ignoring tensor alignment, a block costs

4 + \frac{8}{32} = 4.25 \ \text{bits per weight}

That is the defining geometry of OCP MXFP4. NVIDIA’s NVFP4 also uses E2M1 values, but with an E4M3 scale per 16 values plus a second-level tensor scale, nominally 4.5 bits per weight before that tensor scale. Calling this checkpoint NVFP4 would be wrong, and it would hide the reason a Blackwell-specific kernel cannot simply be selected on an Ampere card. FreeToken names this expert layout ds_fp4; I call it MXFP4 when I mean the numeric format.

Since the A4000 has no FP4 matrix instructions, the portable GPU kernels read packed nibbles and E8M0 scales directly, decode inside the reduction, and accumulate in FP32:

z_n = \sum_k a_k\, v(c_{n,k})\, 2^{e_{n,\lfloor k/32 \rfloor}-127}

Scales load once per 32-weight block and broadcast. A full BF16 copy of an expert is never materialized, which matters more than the arithmetic rate when the budget is this tight. The CPU path reads the same packed row-major banks with no repack. This is a software execution path for an MXFP4 checkpoint rather than native FP4 arithmetic, and the distinction matters for anyone reading the throughput numbers below.

Where the 28% split comes from

The hybrid fetch fraction in Figure 1 was not chosen by hand. It is the FreeToken rule, instantiated with bandwidths measured on this box. Let B be the bytes associated with expert misses in one step, let q be the fraction fetched over PCIe, and let \beta_g and \beta_c be the achieved PCIe gather and CPU expert-compute bandwidths while both paths run concurrently. The two service times are

T_g(q) = \frac{qB}{\beta_g}, \qquad T_c(q) = \frac{(1-q)B}{\beta_c}, \qquad T_{\text{hybrid}}(q) = \max\{T_g(q),\,T_c(q)\}

With ideal overlap the two paths balance at

q^{*} = \frac{\beta_g}{\beta_g + \beta_c} = \frac{9.85}{9.85 + 25.37} = 0.2797

which is the deployed 28%. The measured inputs matter as much as the formula. Both \beta_c = 25.37 GB/s and \beta_g = 9.85 GB/s were measured under contention. Standalone host-to-device bandwidth on this box is 12.3 GB/s, and using that isolated figure would describe a regime hybrid execution never operates in. The formula is portable; these numbers are not. The 28% is also a machine-calibrated starting point derived from a single-rank profile, while serving runs four ranks with two per socket. Sweeping nearby fractions under the real TP=4 workload is the first item on my list.

First touch, or the banks scatter

Linux places an anonymous page on the node of the thread that first writes it. Load a 36.66 GiB rank-local expert bank before binding the rank, and its pages scatter across both sockets; the CPU expert workers then read remote memory, and PCIe DMA may cross the socket interconnect on the way to that rank’s GPU. The Linux distances here are 10 local and 21 remote, so this is not a rounding error.

The runtime now does this, in order, before any large allocation: read the allowed CPU mask and NUMA CPU lists from sysfs; resolve each CUDA device’s PCI bus ID and its local NUMA node; map ranks to their GPU-local nodes; bind rank CPU affinity and record the placement before affinity hides the original topology; allocate and first-touch the banks under that binding; then split the node’s physical cores among ranks with one coordinator reserved per rank. Each rank ended up with nine AVX-512 expert workers and one coordinator.

The placement reporter runs after CUDA graph capture, not before, so what it prints is the serving state rather than an intermediate loader state:

LocationRanksExpert banksGPU allocationCPU
NUMA node 00, 173.31 GiB27.97 GiB20 cores
NUMA node 12, 373.31 GiB27.26 GiB20 cores
Total0–3146.62 GiB55.24 GiB40 cores
Share of counted placement72.6%27.4%
Measured final serving placement. “Counted placement” is host expert banks plus GPU allocations after capture. It is not the checkpoint file size, and it is not a one-to-one disk-to-memory expansion.
Server log showing the 28 percent hybrid fetch fraction, four ranks allocating 61,056 KV tokens each, CUDA graph capture leaving 1.26 GiB free, the profiled DSpark target verify curve from 59.26 ms to 160.32 ms, the acceptance fallback enabled at threshold 60 percent with min-drafted 32 and 64 target-only steps, and a final model distribution reporting 73.31 GiB of host experts per NUMA node, per-rank VRAM of 14.34 and 13.63 GiB, and a total counted placement of CPU 72.6 percent and GPU 27.4 percent
The server printing that same table at the end of startup, which is where the numbers above come from. Reading down: the 28% hybrid fetch fraction is announced, each of the four ranks allocates 61,056 KV tokens, and graph capture runs and leaves 1.26 GiB free. The engine then profiles its own verify curve, 1 rows=59.26ms through 6 rows=160.32ms, where “rows” is the anchor plus k proposals, so one row is width zero. It also prints the fallback thresholds, threshold=60.0%, min-drafted=32, target-only-steps=64, which are the values used in the controller below. The Model distribution (final, NUMA-aware) block is the placement table: 73.31 GiB of host experts on each node, rank 0 at 14.34 GiB of VRAM and ranks 1–3 at 13.63 GiB, nine CPU-MoE workers plus one coordinator each, and a last line summing to CPU 72.6% / GPU 27.4%. This is a 25 August run, so its verify curve sits a couple of milliseconds away from the recorded final configuration charted further down. The shape is the same.

Rank 0 used 14.34 GiB and ranks 1–3 used 13.63 GiB each, so the four cards hold 88.5% of their usable VRAM. The host banks take 29.2% of system RAM. Roughly three quarters of this model lives on the CPU side while it is being served.

Speculative decoding that stays exact

DSpark proposes \gamma = 5 tokens per pass with a drafter that ships inside the checkpoint: three MoE draft blocks, a 128-token sliding window, a low-rank Markov head, and a confidence head. Speculative decoding is worth doing because it is exact. The accepted output has the same distribution as target-only sampling. For a proposal x_i,

a_i = \min\left(1, \frac{p_i(x_i)}{q_i(x_i)}\right), \qquad r_i(x) = \frac{\max\{p_i(x) - q_i(x),\, 0\}}{\sum_z \max\{p_i(z) - q_i(z),\, 0\}}

where r_i is the residual distribution the replacement token is drawn from at the first rejection. Both p and q are formed after the request’s temperature, top-p, and top-k policy is applied. Comparing proposals against target argmax would be faster and would bias a temperature-1.0 request toward the mode, so the implementation does not do that. Proposal probabilities, uniforms, first rejection, residual construction, and bonus sampling all stay on the GPU. Exactly two integers cross to the host per step: the accepted length and the emitted target token.

The acceptance rule was not the hard part. The difficulty is that the target verifies anchor plus proposals before it knows the accepted prefix, so a rejection cannot be implemented as a token-buffer truncate. Every piece of state has an owner and needs an explicit rule:

State ownerCommitRejection or unused suffix
Request token bufferAccepted proposals plus one target tokenTruncate to accepted frontier
Target paged KVKeep accepted rowsReturn unused tail pages immediately
Draft sliding-window KVCatch up from committed target rowsDo not advance from abandoned rows
SWA slotsKeep accepted positionsRelease tail slots immediately
Target auxiliary featuresSelect the accepted physical rowNever commit a rejected feature row
Compressor and indexer carryCommit journal row at accepted frontierRestore the selected row when later speculative rows overwrote the same ring page
DetokenizerEmit each accepted token exactly onceStop at EOS or a stop token inside the block
Client lifecyclePreserve owned committed stateRelease speculative allocations on disconnect or abort
Commit and rejection rules after speculative verification.

The carry journal is the subtle row. Several positions share one 128-token ring page, so releasing whole pages cannot restore an earlier recurrent value inside a page that is still live. The detokenizer row came from a real bug: writing a multi-token block through a single-token path duplicated visible tokens until the contract was fixed to advance engine and tokenizer through the accepted list exactly once.

One more constraint exists only because of tensor parallelism. Every rank must use the same sampling stream and select the same physical graph span. A rank disagreement changes collective ordering and can deadlock the process group, even when each rank’s local token choice looks reasonable on its own.

How many proposals are worth verifying

Let c_i be the confidence, between 0 and 1, that proposal i survives given the previous prefix. Survival through position k and the expected token yield of a width-k verification are

S_k = \prod_{i=1}^{k} c_i, \qquad E(k) = 1 + \sum_{i=1}^{k} S_i

With D the measured draft time and V(k) the median replay time of the target graph for anchor plus k proposals, the one-request scheduler picks

k^{*} = \mathop{\mathrm{argmax}}\limits_{0 \le k \le \gamma} \frac{E(k)}{D + V(k)}

Width zero is legal and still emits one target token; ties keep the smaller width. Measured graph costs are forced monotonic before they enter the selector, so profiling noise cannot make a larger physical graph look cheaper than a smaller one. The measured values:

Bar chart of target verify time V(k) rising from 61.17 ms at width 0 to 157.79 ms at width 5, against a constant 23.47 ms draft cost, with the resulting 33.1 tokens per second ceiling annotated
The measured cost of one speculative cycle. The blue bars are target verify time V(k), rising from 61.17 ms for the anchor alone to 157.79 ms for all five proposals. The rise is sublinear, which is why verifying several positions in one pass is worth doing at all. The green bars are the 23.47 ms the drafter costs regardless of how many of its proposals survive. Every cycle pays one green bar plus one blue bar and emits between one and six tokens. The annotation is the best case those two numbers allow, and all of it is specific to this machine and this cache state.

The curve also sets the throughput ceiling. With full five-proposal acceptance a cycle emits six tokens:

R_{\max} = \frac{6}{(23.47 + 157.79)\times 10^{-3}} = 33.1\ \text{tok/s}

and at the 84% acceptance I see on code, a cycle emits 1 + 5(0.84) = 5.2 tokens:

R_{0.84} \approx \frac{5.2}{0.18126} = 28.7\ \text{tok/s}

which matches the 28–30 tok/s region I measure. No amount of confidence-threshold tuning gets this machine to 60 tok/s. Only a substantial cut in draft cost or target verify cost does.

When the drafter stops paying for itself

The width selector runs after the drafter has already executed, so it cannot recover the 23.47 ms of draft cost on a request that keeps rejecting proposals. Structured code is locally predictable and accepts well. Open-ended prose and hard reasoning often do not. A workstation serving one user should not keep paying the draft cost just because the prompt was initially eligible.

The controller watches acceptance instead of guessing from the prompt text. Over a request-local window, with A accepted draft tokens out of G verified proposals, measured acceptance is

\widehat{\alpha} = \frac{A}{G}

Once the window holds at least m verified proposals, the controller sends the request to target-only for C steps when \widehat{\alpha} \lt \tau, and leaves it on DSpark when \widehat{\alpha} \ge \tau. The evaluated settings are \tau = 0.60, m = 32 and C = 64.

State machine: DSpark mode switches to target-only mode when at least 32 proposals are observed and measured acceptance falls below 0.60, and returns after 64 target steps for a re-entry probe
The acceptance-guided fallback as a two-state machine. The gold arrow going right is the trip condition: once at least 32 proposals have been verified and fewer than 60% of them were accepted, the request stops drafting and runs the target alone. The blue arrow coming back is the re-entry: after exactly 64 target-only steps the controller probes speculation again and starts a fresh measurement window, so a request that becomes predictable later is not stuck in target-only mode for the rest of its life. The dashed note is what makes re-entry useful rather than merely cheap. Both states sample from the correct distribution, so a request can cross between them mid-generation without changing what it would have produced. The controller is disabled by default in the submitted code.

Both modes are target-correct, so switching between them preserves the requested sampling distribution. When DSpark runs, the acceptance and residual rules above are untouched; when target-only runs, tokens are drawn straight from the target distribution. New request IDs reset every counter, so one request can never inherit another’s decision.

Server log of a Spanish-language creative writing request in which the DSpark fallback repeatedly reports acceptance between 37.5 and 57.6 percent of 32 proposals, switches to target-only for 64 steps, then probes speculation again, while decode throughput oscillates between about 15 and 18.75 tokens per second
The controller running on the workload it was built for: an open-ended Spanish-language story, visible as faint text behind the log. Two lines alternate. request 0 accepted 53.1% of 32 proposals; target-only for 64 steps is the trip, a full window measured under the 60% threshold, so drafting stops. request 0 probing speculation after 64 target-only steps is the re-entry probe. It trips again at 46.9%, 50.0%, 37.5%, then 57.6%, which is what an unpredictable request looks like: the drafter never becomes profitable, and the controller keeps declining to pay for it. Cumulative acceptance on the right climbs from 34% to 53% as target-only steps dilute the counter, and generation throughput oscillates between roughly 15 and 18.75 tok/s, the mixed and target-only windows from the results table.

The composition has four properties: request-local measured acceptance, a bounded target-only interval, drafter-state maintenance during that interval, and re-entry probes. I make no priority claim over other speculative decoders. The checkable claim is that this controller was added to FreeToken’s DeepSeek-V4 DSpark path, tested against its hybrid state machinery, and measured on a workload where the drafter was persistently unprofitable.

Results

Three reporting rules apply to these numbers. Prefill and decode throughput are reported separately, since a short prompt cannot exercise the 2,048-token prefill path. The first status interval after idle time is excluded from steady-state decode, because it contains idle and calibration time. Speculative counters are cumulative since server start, so per-request acceptance is the difference of the counters immediately before and after the request.

ConfigurationWorkloadAcceptanceThroughput
Target-only baselineReference decoden/aabout 18 tok/s
DSpark PR stackStructured code65–70%27.7–29.7 tok/s (1.54–1.65×)
Enhanced-v2 + 2048Structured code control81.6%29.29, 30.08 tok/s
Enhanced-v2 + 204825-horses reasoning61.0%18.37 tok/s mean
Fallback enabledStructured code controlhigh; no useful trip30.55, 27.33 tok/s
Fallback disabledCreative proselowabout 15.7 tok/s end-to-end
Fallback enabledSame creative prompt30.3–59.4% windows17.18 tok/s end-to-end
Single-request generation, TP=4, temperature 1.0. Rows from different prompts are not replicates and should not be compared as though they were.
Server log of a baseline run without DSpark, showing per-rank CPU MoE pools of nine AVX-512 threads, the auto-resolved 28 percent hybrid PCIe fetch fraction, KV cache allocation, CUDA graph capture with 0.94 GiB free, and decode batches reaching 13.41 to 14.69 tokens per second
Before, on 22 August: four rank-local CPU MoE pools of nine AVX-512 threads each, the 28% hybrid fetch fraction resolved from the bandwidth benchmark, and decode climbing to 14.64–14.69 tok/s. This early configuration is slower than the approximately 18 tok/s target-only baseline used in the PR-3 comparison. It is the same machine before the DSpark path worked at all.
Server log showing DSpark adaptive verify lines with stale confidence vectors and selected widths from 0 of 5 to 5 of 5, per-cycle spec timing with draft around 21 ms and target CUDA time from 36 to 145 ms, and decode batches at 28 to 32 tokens per second with 82 to 83 percent of proposals accepted
After, later the same day: the selector choosing 0/5, 1/5, 4/5 and 5/5 from the stale confidence vector on the left, cycle wall time moving with it from 70 to 179 ms, and decode settling at 28–32 tok/s with 82–83% of proposals accepted. This is the selector from the chart above, running.

The structured-code improvement is the primary gain: 27.7–29.7 tok/s against an approximately 18 tok/s target-only baseline, with later controls reaching about 30 tok/s at higher acceptance. The fallback result is narrower. One matched creative request improved from about 15.7 to 17.18 tok/s, a 9.4% rate increase, with wall time falling from about 98 s to 88.07 s for 1,513 tokens. During fallback the target-only windows reached 17.86–19.59 tok/s. That is one matched run, with the baseline wall time inferred from server timestamps and slightly different natural completion lengths. It is evidence for the controller, not a workload-wide average.

The largest confirmed wall-time improvement came from doubling the default prefill chunk.

Chunk sizeFull-chunk throughputTotal prefill wall time
1,024 tokens321.8–323.2 tok/s25.79 s
2,048 tokens625.4–627.3 tok/s15.31 s
Changeabout 1.94×40.6% lower
Same fixed 7,924-token prompt on both arms.

That gain comes from denser prefill work and better amortization, not from the fallback. A separate 9,139-token prompt with a 200-token response completed coherently without OOM. A faster chunk is worthless if it crosses the workstation’s graph or activation budget.

The results that did not work

Graphing the drafter did essentially nothing. A matched A/B on the same commit put the captured drafter backbone at 25.03 tok/s end-to-end against 24.78 eager, and the steady mean favored eager, 24.23 against 23.97. Capture reduced the drafter substage from about 21.71 to 18.43 ms, but jitter rose from 3.72 to 4.84 tok/s standard deviation, and fallback entries rose from 7 to 12. Both outputs were coherent. The deployed service keeps the eager drafter, with graph replay behind an opt-out switch until repeated tests explain the variance.

More cache is not monotonically better. A 0.90 memory ratio auto-sized 2,509 MoE cache slots and left 1.02 GiB before graph capture, at which point DSpark verification capture failed with OOM. A 0.80 ratio selected 2,016 slots, left 2.56 GiB before capture and 1.26 GiB after, and started. Residency competes directly with the reserve that graph capture needs.

Reasoning termination is still a real failure. Bounded bridge-and-torch prompts found the correct strategy without repetition, then consumed a 1,200-token budget formalizing the proof and never entered the final answer channel. A 25-horses run found the correct seven-race strategy and was cut during its lower bound. Correct intermediate reasoning that never reaches an answer is a serving-quality bug, and output budgets and termination behavior need their own fix.

Concurrency is not claimed at all. Adaptive width selection here is specialized to one active request. Padding every request to a single maximum width is not equivalent to a global capacity decision across flattened variable-length requests, so there is no multi-request throughput number in this work.

Making it a service

The model did not become a service when the first correct token appeared. Some of what stood between those two states:

  • Compiler. The workstation’s default GCC 16 was newer than the CUDA JIT would accept, so the service pins NVCC and host C++ to GCC/G++ 15.
  • KV capacity. The validated deployment allocates 477 full-KV pages, or 61,056 tokens of prompt plus output. The architecture supports far more context than that. An architecture limit and an allocated workstation capacity are different quantities and should not be quoted interchangeably.
  • Loading and rendezvous. Expert loading is serial under the proven profile, and the distributed startup timeout is 1,800 s to tolerate legitimately skewed rank-local bank loading.
  • Readiness. The HTTP frontend binds before the expert banks finish loading, so the manager waits for the engine’s ready to serve record instead of treating an HTTP response as model readiness.
  • Teardown. TP workers are detached multiprocessing children and can retain the rendezvous port after the frontend stops. The management script resolves only service-specific ranks and shared-memory objects, and never terminates by broad process-name match.
  • Provenance. Source changes reach the Lenovo checkout only through a named branch pushed to the fork, then fetch, switch or fast-forward, then server-side tests at that exact commit. Copying source files directly into the server is outside the validated procedure.

These items are part of the feasibility result. A model that starts only after manual process cleanup, or that OOMs on a long prompt right after a clean benchmark, is not a reproducible deployment.

The upstream stack

Three stacked PRs against FlashML-org/FreeToken, reviewed in the order 1/3, 2/3, 3/3 even though GitHub numbered the second layer #69 and the foundation #70. These are cumulative diffs, so a stacked PR includes its prerequisites and the rows must not be summed:

OrderPull requestHeadCommitsFilesAdded / removed
1/3#70, TP runtimec066b38618903 / 121
2/3#69, exact DSpark2cf938b8535,360 / 193
3/3#71, adaptive verification4800af018737,465 / 335
Cumulative diff statistics as inspected. All three were open at inspection time and are being rebased under review, so heads may move.

PR 1 is deliberately independent of DSpark. Its only job is to make one target-model forward mathematically complete and memory-feasible across four ranks. PR 2 loads the checkpoint drafter, builds draft context KV, verifies a block, and implements exact greedy and probabilistic acceptance, then fixes what only appears under full hybrid serving. PR 3 is where the correct implementation becomes a measured runtime: adaptive TP verification, GPU-resident probabilistic sampling, exact multi-token detokenization, fused hyper-connection normalization, long-prefill bounds, in-place NCCL, placement reporting, route-aware expert fetch, and the fallback.

Two GPU-side changes in PR 3 need credit and caveats. The fused hyper-connection pre-normalization reads the BF16 source once and accumulates squares in FP32, instead of materializing a full FP32 copy plus a second FP32 square tensor. On an RTX 6000 Ada microbenchmark at the DSV4 hidden geometry, the T = 8192 operation fell from 1.944 to 0.304 ms, a 6.4× kernel speedup. That optimization originated with Gabriel Devenyi’s FreeToken PR #101; my change extends it to the DSpark drafter and adds long-prefill bounds. The Ada microbenchmark is not an A4000 end-to-end result. Separately, TP all-reduce now operates in place on its contiguous input, removing the two device copies the older symmetric-staging path performed around every reduction. That path had dominated an earlier TP=4 verification trace on this PCIe configuration.

At submission the complete three-PR stack passed 192 focused DSV4 tests on the TP=4 server, and the fallback lineage passed 202 DSpark and hybrid-fetch tests at its exact pushed commit. Coverage runs from TP=1/2/4 shape ownership and quantization alignment, through greedy and sampled acceptance including residual recovery and width zero, to paged KV, SWA, recurrent carry, and client-disconnect cleanup.

What made it feasible

No single optimization explains this. MXFP4 shrinks expert storage, and the checkpoint still exceeds aggregate VRAM. CPU offload adds capacity, and a single PCIe expert miss can dominate a short decode step. Tensor parallelism adds VRAM, and it also adds collectives and creates four host-memory consumers. DSpark amortizes target work, and low acceptance can make its draft cost counterproductive. NUMA binding improves locality, but only if it happens before first touch. CUDA graphs cut launch overhead, but only if fixed buffers and state ownership survive replay.

The feasibility chain is conjunctive:

\begin{aligned}\mathcal{F} = \ &\text{rank-local storage} + \text{MXFP4 execution} + \text{NUMA-local banks} \\ &+\ \text{hybrid miss service} + \text{exact speculative state} + \text{bounded graph memory}\end{aligned}

Remove any one term and you either break correctness, exceed memory, or leave a large avoidable latency on the table. Each of those ideas has prior work behind it: tensor parallelism, speculative sampling, paged state, microscaling formats, offload, CUDA graphs, NUMA placement, and the bandwidth-adaptive MoE execution that FreeToken contributes. DSpark supplies the trained drafter and the confidence-scheduled verification. The contribution is at the integration boundary, where these mechanisms have to agree about tensor ownership, quantization metadata, speculative state, rank-local memory, PCIe service, graph replay, and request lifecycle.

The scope is one machine, one active request, limited repetitions, and qualitative output checks rather than benchmark accuracy or human preference studies. The bandwidth split formula is portable; its measured inputs are not. The case study shows something narrower than a speed number: a 156 GiB MXFP4 checkpoint becomes practical on 62.40 GiB of Ampere VRAM when memory ownership, numeric format, speculative correctness, PCIe service, and NUMA locality are designed together rather than separately.

The full manuscript, with the complete ownership table, the threats to validity, and the reproduction checklist, is here as a PDF.

References

S. Yang, X. Fan, M. Pan, H. Xi, Z. Wang, S. Sun, K. Keutzer, S. Han, M. Zaharia, C. Xu, and I. Stoica, “FreeToken: Efficient edge-native MoE serving with bandwidth-adaptive execution,” arXiv preprint arXiv:2608.16157, 2026. [Online]. Available: https://arxiv.org/abs/2608.16157

Posted on

Thirteen Bugs, Six Layers: Stabilizing DeepSeek-V4 Flash 0731 on Two DGX Sparks

The DeepSeek whale logo on a screen behind a phone showing the NVIDIA logo

The two NVIDIA DGX Spark nodes arrived on August 10. Since then I've been running deepseek-ai/DeepSeek-V4-Flash-0731 on them, on a plain-venv port of vLLM 0.27.1 with CUDA 13.3, working toward an eventual upstream PR that makes this pairing work out of the box. Speed was never the problem. The port hit 64.6 tok/s on coding prompts and 69–70 tok/s on HTML generation in the first day, with 78–89% draft acceptance against the model’s own speculative head. Correctness took the six days after that. By the time the stack settled down I had found thirteen distinct bugs spread across six layers of the serving stack. This post covers all thirteen, and goes deep on the five that took the longest to find, with the math behind each failure and each fix.

The DeepSeek whale logo with the text DeepSeek V4 Flash 0731

The model, briefly

DeepSeek-V4-Flash-0731 is a mixture-of-experts model: 284B total parameters, 13B active per token. That’s a sparsity ratio of

\frac{P_{\text{active}}}{P_{\text{total}}} = \frac{13\text{B}}{284\text{B}} \approx 4.6\%

It pairs a hybrid sparse-attention stack (Compressed Sparse Attention plus Heavily Compressed Attention) with Manifold-Constrained Hyper-Connections, and DeepSeek reports that combination reaching 27% of V3.2’s per-token inference FLOPs and 10% of its KV-cache footprint at 1M context. Expert weights ship in MXFP4, attention/norm/router parameters stay in FP8. A Lightning/DSA indexer picks a sparse set of attention positions per token instead of attending densely. That’s the mechanism behind the 10% KV number, and it’s also where two of the bugs below turned out to live.

Only the 0731 checkpoint matters here (sha 7872f01b, dated 2026-08-01, 166.9 GB). Two lookalikes trip people up, and one of them tripped me up too, because a third-party recipe’s .env file defaults to it: deepseek-ai/DeepSeek-V4-Flash is an earlier preview whose README opens with “We present a preview version,” and deepseek-ai/DeepSeek-V4-Flash-DSpark is dated 2026-07-04, four weeks older. All three share an architecture, so shard counts and file sizes look almost identical; only the checkpoint date and the README tell them apart. 0731 already ships the DSpark/MTP draft head as part of the base checkpoint (72,317 tensors, 4,705 of them mtp.*), so there’s no reason to layer the DSpark repo on top of it.

Why it's worth the effort

Thirteen bugs is a lot to spend on one model, so let me be specific about why this one earned it. In DeepSeek’s own release benchmarks, V4-Flash-0731 (the 13B-active checkpoint this whole post is about) beats GLM-5.2 on most of the agentic and coding benchmarks that matter for this kind of use: Terminal Bench 2.1, 82.7 vs 81.0. Cybergym, 76.7 (GLM-5.2 wasn’t reported). DeepSWE, 54.4 vs 46.2. Toolathlon-Verified, 70.3 vs 59.9. Agents’ Last Exam, 25.2 vs 23.8. AutomationBench, 25.1 vs 12.9. Both DSBench splits, 68.7 vs 61.8 on full-stack and 59.6 vs 54.5 on hard.

Benchmark table comparing DeepSeek-V4-Pro-0813, DeepSeek-V4-Flash-0731, their preview versions, GLM-5.2, Kimi-K3, Opus-4.8, and Fable 5 across HLE, Terminal Bench, NL2Repo, Cybergym, DeepSWE, Toolathlon, Agents' Last Exam, AutomationBench, and DSBench
Self-reported by DeepSeek at release. I haven’t independently reproduced these numbers, only V4-Flash-0731’s serving behavior on the Sparks.

It trails Kimi-K3, Opus-4.8, and Fable 5 on most rows, and it trails its own much larger sibling V4-Pro-0813 everywhere. That’s not surprising for a model running at roughly 4.6% of the active parameters (the ratio from the section above). The narrower point, and the one that actually matters for two DGX Sparks, is this: at the active-parameter budget two 128 GB unified-memory boxes can hold and run at a livable speed, this is about as good as it currently gets. It’s not a compromise pick. That’s why a week chasing thirteen bugs made more sense than settling for a smaller model that would have needed none of them.

The hardware and the port

The two Sparks (GB10, sm_121a) are connected directly with two QSFP DAC cables into the ConnectX-7 NIC’s two cages, running tensor-parallel size 2 across the pair. One cable alone already gets close to line rate, about 196 Gb/s aggregate across its two PCIe partitions. The second cable only adds about 13% more, up to roughly 221 Gb/s. I keep both plugged in anyway. The real reason is redundancy and link stability, not the extra bandwidth. Getting either cable up to its rated 112 Gb/s per partition took one non-obvious step: if you cable the QSFP ports while a node is already booted, the NIC firmware misreports PCIe slot power and throttles the link to about 13 Gb/s no matter what you do with message size or queue depth. The fix is just to reboot with the cables already seated.

Two NVIDIA DGX Spark units resting on their shipping boxes on a desk

On top of that fabric, I ported an earlier 0.21-RC container stack to a plain venv on vLLM 0.27.1: Marlin MXFP4 MoE, FULL_AND_PIECEWISE CUDA graphs, and later CUDA 13.3 installed side by side with 13.0. The driver stays at 580.173.02 on purpose. A driver bump broke GB10 detection for other people on this hardware, and nvidia-smi still reporting “CUDA 13.0” afterward just means that’s the driver’s max-supported version, not that the upgrade failed. The port-0271 branch on my vLLM fork, based on the v0.27.1 tag, is the staging ground for a PR I want to eventually get onto vLLM main. That branch is where all thirteen fixes below actually live.

Two NVIDIA DGX Spark units facing each other on a desk, joined by a QSFP cable
The two Sparks, joined by the QSFP link that carries tensor parallelism across the pair.

Thirteen findings, six layers

Before I go deep on any single bug, here’s the full inventory. No one layer of the stack was responsible, and where these things hid turned out to be a finding in itself. Ordered by where each one sits on the path from a cold boot to a served token:

\begin{array}{c} \boxed{\text{1. Config \& CLI resolution, kernel-bucket dispatch}\ \ (\#1,\#2,\#3,\#4,\#5)}\\[4pt] \downarrow\\[4pt] \boxed{\text{2. Platform introspection \& persistent adaptive-profile cache}\ \ (\#6,\#7,\#8)}\\[4pt] \downarrow\\[4pt] \boxed{\text{3. KV-cache dtype \& packed-layout definition}\ \ (\#9)}\\[4pt] \downarrow\\[4pt] \boxed{\text{4. KV block allocator}\ \ (\#12)}\\[4pt] \downarrow\\[4pt] \boxed{\text{5. Sparse-attention metadata builders: indexer, SWA, C128A}\ \ (\#10,\#11)}\\[4pt] \downarrow\\[4pt] \boxed{\text{6. CUDA-graph capture \& replay}\ \ (\#13)} \end{array}

Layer 1 showed up during bring-up, before the model ever answered a real request, and every bug there was a hard crash or a deterministic hang. Annoying, but honest: the thing just stopped. Everything from layer 3 down only showed up under real load, after the server had been running correctly for hours, and every one of those bugs was a silent wrong answer instead of a crash. That’s the pattern for the whole week: loud failures early, quiet ones late.

Layer 1: getting the model to boot at all

  • #1, block-size dispatch gate. vLLM’s SM120 decode kernels only dispatch when the packed-KV page block size is exactly 64 (_DECODE_DSV4_PAGE_BLOCK_SIZE = 64, heads 8–128, top-k in {128, 512, 1024}). The container-inherited --block-size 256 failed that check for small batches, which fell through to a prefill-only orchestrator and crashed warmup with Check failed: num_tokens > 64 (5 vs. 64): Decode must go through sparse_mla_sm120_decode_dsv3_2/dsv4.
  • #2, a ZeroDivisionError from switching to block 64. DSv4’s compressed-MLA layers carry a compress_ratio of 128 in the checkpoint, so storage_block_size = 64 // 128 comes out to zero. Block 64 was the wrong lever in the first place: the sliding-window cache already hardcodes block size 64 upstream. The real fix left the main store at 256 and let SWA keep its own 64.
  • #3, a top-k value with no dispatch bucket. The DSpark draft layer called the decode kernel with a top-k of 256, and the SM120 dispatch table only has buckets at 128, 512, and 1024. Fix: pad the index tensor up to the next bucket with -1, a value the kernel already knows to skip.
  • #4, a warmup exception that turned into a permanent hang instead of an error. Padding every call unconditionally broke batches over 64 tokens, which take a different orchestrator path and reject the padded shape. That raised an exception inside warmup/capture, and the engine’s inter-process broadcast just swallowed it as an idle wait instead of propagating it. Two hangs that looked separate turned out to be this same bug, reproduced offline with a 33-case kernel sweep. Fix: only pad calls that were already decode-sized.
  • #5, KV pool undersized for the advertised context. At the default memory utilization, PIECEWISE graph accounting left the KV pool about 0.14 GiB short of what 1M context needs. Fix: raise --gpu-memory-utilization.

Layer 2: the machine lying to the software about itself

  • #6, GB10’s unified memory is invisible to NVML. get_device_total_memory() raised NVMLError_NotSupported. Falling back to PyTorch’s own memory query, the two otherwise identical Sparks reported 130,663,235,584 and 130,663,231,488 bytes, a 4 KiB difference that had to be normalized away (both ranks round to 124,610 MiB) before the two nodes could agree on anything downstream.
  • #7, a nested list mistaken for a flat one. runner.attn_groups is list[list[AttentionGroup]], and code that iterated it as a flat list crashed with 'list' object has no attribute 'backend'. Fix: flatten before use.
  • #8, a persistent cache keyed on its own volatile output. An adaptive startup-profile cache, meant to skip the expensive graph/profile phase on a warm restart, accidentally included the resolved KV-allocation layout (block count, offsets, buffer sizes) inside its own cache key. That layout is intentionally nondeterministic across boots: one cold run allocated 1,399,106 KV tokens, an otherwise identical restart allocated 1,429,947. So the cache missed on every single restart and silently re-ran the full profile it was supposed to skip. Fix: pull just the volatile allocation object out of the key and keep everything else (KV specs, dtype, block size, model config, hardware, kernels, graph sizes). A regression test now checks the fingerprint stays the same when the simulated allocation is doubled.

The remaining five findings are the ones that took the longest to find, and the ones that mattered most, because none of them crashed anything: the KV-cache dtype that turned out to be mislabeled, the indexer stall, the tiering disagreement, the block-zeroing stride bug, and the CUDA-graph stride bug. The symptoms ranged from full gibberish and bursts of non-Latin characters, to raw unformed tool-call markup leaking into visible content, down to replies that read as completely coherent and were just quietly wrong. That last one is the dangerous mode. This model is strong and precise when the stack underneath it is healthy, so a degraded reply doesn’t look degraded. It looks like an ordinary answer that happens to be wrong. Each of the five gets its own section below, with the address-level math of what was actually happening in memory.

Finding #9: a KV-cache dtype that is not what it says

On this SM121 sparse-MLA path only two KV-cache dtypes actually work: fp8_ds_mla and nvfp4_ds_mla. bf16/fp16 hit a hard assertion, "DeepseekV4 fp8_ds_mla layout only supports fp8 kv-cache, got bfloat16", because the SM121 sparse-MLA kernels only exist for uint8-packed layouts.

The first corruption I chased looked like a precision problem. fp8_ds_mla was clean at short context (70.2 tok/s, 85–89% draft acceptance), but at 40K–77K tokens it started injecting stray “#” tokens mid-sentence, which escalated into full collapse: code fragments, phrase loops, language mixing. The metrics were unambiguous. Draft acceptance crashed to 6–10%, with rejection positions 2 through 5 sitting at exactly 0.000, meaning the draft head and the target model had stopped agreeing entirely. An A/B against nvfp4_ds_mla with CUDA 13.3 held constant came back clean, 4 probes out of 4. That looked like enough evidence, so I reverted to nvfp4_ds_mla in production and wrote it up as a precision regression in the fp8 sparse-MLA decode path.

It wasn’t enough evidence. A later audit of deepseek_v4/attention.py found that nvfp4_ds_mla maps onto the exact same uint8, 584-bytes-per-token packed layout as fp8_ds_mla. Same UE8M0-FP8 quantization kernel writing it, same FlashInfer SM120 kernels reading it. The only runtime difference between the two labels is page alignment:

\text{bytes/token: main}=584,\ \text{SWA}=512,\ \text{indexer}=512 \quad(\texttt{fp8\_ds\_mla}) \text{vs. alignment}=576 \text{ everywhere} \quad(\texttt{nvfp4\_ds\_mla})

Same bit format on both arms of the A/B. The four-probe comparison wasn’t a precision test at all, and it ran before finding #12 below was known, with concurrent traffic present. So the result was confounded by a bug that had nothing to do with KV dtype. The lesson goes beyond this one flag: a name on a KV-cache option is a claim about behavior, not a guarantee of it. The only way to know which bytes are actually on the wire is to read the kernel that writes them.

Finding #10: a stalled indexer at very long context

The DSA indexer scores candidate attention positions with an MQA-logits computation. DeepGEMM has no arch-12 kernels for sm_121a, so this step falls back to a pure PyTorch implementation, _fp8_mqa_logits_torch. At roughly 450K+ prompt tokens, that fallback pinned a worker inside itself for over 30 minutes. GPU utilization sat at 96%, the engine’s own status logs went quiet, and the HTTP API kept answering, which makes it look like a hang from the outside when it’s actually a very slow path grinding to completion. py-spy sampling of the stuck worker is what actually surfaced this. Nothing in the request-level logs told it apart from a genuine deadlock.

The actual bug was a dispatch error. Requests with clean_logits=False, a legitimate and common mode, got routed into the slow torch fallback instead of the working fp8_mqa_logits_triton kernel that should have handled them. The fix was a one-line dispatch correction plus a regression test that checks the routing. I validated it against a 468,540-token needle search that had never completed before: 498 seconds, correct answer. This went upstream as a cross-fork PR into jasl’s #41834 branch, plus a field-report comment on vllm-project/vllm#41063, citing three ROCm issues (#48576, #41963, #52109) as likely the same silent-fallback problem on a different vendor’s SM12x-equivalent path.

Finding #11: three sparse-attention builders disagreeing about one boundary

Sparse-MLA on this port runs across three builders: the indexer, the sliding-window attention path, and the main sparse-MLA path. All three read the same topk_indices_buffer, sliced at a boundary each one computes on its own: num_decode_tokens. The port had cherry-picked a fix, sparse_short_extend_tiering(), for how that boundary should treat a “short extend” (a prefilling row with six tokens or fewer), but it only got wired into the indexer. The other two builders stayed on the old default of always treating short extends as decodes.

Under an ordinary batch, all three agree. Under a batch that mixes a normal prefilling row with a short-extend row, they don’t: the indexer computes one value of num_decode_tokens, the other two compute a different one, and the co-scheduled prefill chunk ends up attending to positions shifted by that disagreement. The wrong attention output for that chunk gets written into later layers. Because it looks like a plausible continuation instead of garbage, it gets written into the prefix cache and stays wrong until the process restarts and the cache is gone with it. That’s exactly the signature of “the model loops after hours, and a restart fixes it.” Given how the DSpark draft group schedules things, this only comes up from budget-starved chunks with six tokens or fewer left in an 8,192-token budget, which puts the estimated collision rate around 0.07% per prefill. Low frequency, but a real defect. The fix was a single upstream commit that aligns all three builders’ tiering logic, cherry-picked onto the four active fork branches.

Finding #12: the real root cause, a stride bug in the block zeroer

This turned out to be the bug the fp8-vs-nvfp4 A/B was actually chasing, without anyone knowing it. DeepSeek-V4’s packed KV layout stores, for each physical block index b, one contiguous segment per attention layer \ell \in \{0,\dots,L-1\}, each segment s bytes wide. So the full row width for one block is

S \;=\; \texttt{kv.stride(0)} \;=\; L \cdot s

vLLM 0.27.1’s KVBlockZeroer (vllm/v1/worker/utils.py), when it allocates a fresh block for a specific layer \ell, is supposed to clear only that layer’s own slice, \big[\,bS + \ell s,\ bS + (\ell{+}1)s\,\big). Instead it took its zero-fill length from kv.stride(0) itself, the width of the entire multi-layer row, and wrote S bytes starting from that same offset. Here’s the address line, with the layers of block b correctly zeroed on the left and the one bad write on the right:

\underbrace{[\ell{=}0]\ [\ell{=}1]\ \cdots\ [\ell{=}L{-}2]}_{\text{zeroed correctly, one segment }s\text{ at a time}}\ \ \overbrace{[\ell{=}L{-}1\text{ of block }b]\ \ [\ell{=}0\text{ of block }b{+}1]}^{\text{one write of length }S\text{ starting at }bS+(L-1)s}

Because the write length S is wider than the single segment s it was supposed to cover, zeroing the last layer of block b runs S-s bytes past the true end of block b’s row and wipes out nearly the entire first segment of the next physical block, b+1. This only bites when three or more requests prefill at the same time: a fresh block index b gets allocated to one request right when block b+1, already written, belongs to another live request whose data quietly vanishes.

The symptom matches “coherent but wrong” almost exactly. One 256-token block in an otherwise fine document answers incorrectly (a plausible fabrication, not noise), its neighbors are unaffected, the wrong block sticks around because it’s living in the prefix cache, and a restart clears it because the cache goes with it. The reproduction numbers make the concurrency threshold obvious: 8 concurrent documents corrupted 6 of 7 shared blocks, 4 documents corrupted 3 of 4, 2 documents corrupted 0 of 2 (below the 3-way threshold), and a single document alone was always clean. Capping max_num_seqs to 2 as a stopgap brought a 4-document run down to 0 corrupt. The real fix was cherry-picking three upstream commits merged after the 0.27.1 cut (#50276, #51749, #52058) onto all four fork branches. Afterward, the same 4-document run stayed at 0 of 4 with max_num_seqs back at 12, and an 8-concurrent run with sidecar decode traffic came back 0 of 8 corrupt over 714 block reads.

Finding #13: a second stride bug, this time between capture and replay

The last bug only showed up on an instance that had been serving for a while. After 17 hours warm, the same temperature-0 question against the same cached KV gave soft, visibly wrong logits when the request ran alone (lp ≈ -0.74 and -1.49 on the top two candidates, drifting a bit run to run), and sharp, correct logits (lp ≈ -0.000) whenever any neighboring request shared the batch. Right after a fresh restart, the alone case was sharp too. Sampling parameters didn’t matter here; the argmax itself was wrong, not just the temperature. On the same degraded instance, a plain 1K-token solo request eventually crashed the worker with an indexSelectSmallIndex device assertion. Because the process exited with status 0 instead of a failure code, systemd’s Restart=on-failure never kicked in.

A targeted trigger reproduced the exact corruption seen in real usage: raw <|DSML|tool_calls> markers leaking into visible content, language mixing, code fragments, bursts of “#”. The trigger was three concurrent ~44K-token thinking requests plus one short request. The same three long requests without the short one stayed clean, which pointed at something in how the decode kernel handles rows across a mixed batch, not at long context on its own.

The root cause was a second stride mismatch, this time between CUDA-graph capture and runtime replay in the C128A decode path. sparse_mla.py computes active_topk_width from the live batch’s max sequence length (128, 256, or 512 for an ordinary batch), but the FULL-cudagraph capture for this kernel fixes its row stride once, at capture time, using max_model_len, which works out to 8,192. Let t index a row within a decode step’s batch, where t=0 is the accepted token and t \ge 1 are the five speculative rows from the k=5 MTP draft head. Here’s the captured kernel and the runtime builder side by side for the first few rows:

\begin{array}{c|ccccc} t & 0 & 1 & 2 & 3 & 4\\ \hline \text{addr}_{\text{capture}}(t) = t\cdot 8192 & 0 & 8192 & 16384 & 24576 & 32768\\ \text{addr}_{\text{runtime}}(t) = t\cdot w,\ \ w{=}128 & 0 & 128 & 256 & 384 & 512 \end{array}

Row t=0 lines up by coincidence. Every draft row t \ge 1 gets read by the captured kernel from an address the runtime builder never wrote on this step, because w never equals 8192 for any batch shape actually seen after warmup. What the captured kernel finds there instead is whatever the previous replay of that same buffer left behind. Right after a fresh capture that memory is still -1, a harmless placeholder, so a brand-new instance passes every guard. But once a later batch has legitimately written past offset 8192, which takes concurrent long-context traffic (hence “only after hours warm”), those stale bytes belong to another request’s compressed-KV slot ids. That one mismatch explains every symptom above. Soft and wrong when run alone, because the stale bytes are just noise. Sharp when run with a neighbor, because only the alone path replays the polluted batch-1 graph. The crash, because a NaN logits row sent the draft head’s embedding gather to an out-of-range index. The language mixing, because the stale bytes were literally another conversation’s identity leaking into this one’s decode step.

The fix pins active_topk_width to the same capture-time constant the graph was built with (c128a_max_compressed) instead of recomputing it from batch shape, which closes the gap between \text{addr}_{\text{capture}} and \text{addr}_{\text{runtime}} for every t. It shipped with a regression test that fails on the unfixed tree, and it’s cherry-picked onto all four active fork branches. Validation: three solo probes came back sharp, 12 of 12 at lp = -0.000. Two full rounds of the exact trigger pattern (three 44K-token thinking requests plus one short request, run four-way concurrent) came back clean 12 of 12, no DSML leakage, no non-Latin corruption. And a log fingerprint check across every request since the restart showed only the two legitimate stride widths, 512 and 8,192, ever appearing. The mismatch this bug depended on has no path left to occur.

What actually gates a change now

None of this would have been findable without holding one variable constant at a time, and without treating a clean four-probe A/B as the start of an investigation instead of the end of one. A few concrete practices came out of the week:

  • Guard 4, concurrent-prefill KV block integrity: three or more documents prefilled together, then a per-block read-back against the shared prefix cache. This is the only test shape that ever caught the block-zeroer bug (finding #12). Anything at two concurrent requests or below stayed clean by construction.
  • Guard 5, a margin probe run both alone and with a neighboring request. The C128A stride bug (finding #13) was invisible to any test that only checked the alone case right after restart and never came back to it hours later, or that only ever checked the neighbored case.
  • A needle ladder up to a three-needle 44K-token probe, an acceptance canary on draft/target agreement, and a puzzle-solving smoke test, all in quality-guards.md, run after any change to vLLM, CUDA, or drivers.
  • A reusable e2e-battery (quick/full) covering the same ground end to end, always launched detached (setsid nohup, its own log file) so an interrupted session can’t take a running load test down with it.

Two operational details cost real debugging time before I understood them, so they’re worth writing down. The systemd unit for this service, vllm-deepseek-mxfp4@, runs with Restart=on-failure. A manual pkill and relaunch gets raced by systemd bringing the old configuration back about 60 seconds later, so every restart during an experiment has to go through systemctl, never a bare kill and relaunch. And GB10 exposes no wattage cap at all: nvidia-smi -pl is unsupported and there’s no ACPI power limit. So the only thermal lever available is an SM clock cap, running as a boot-enabled service on both nodes.

Where this goes next

port-0271 stays the staging branch. The goal is a PR to vLLM main that makes DeepSeek-V4 Flash work on DGX Spark out of the box, written to upstream standards from the start: debug and instrumentation commits kept isolated so they drop cleanly on rebase, each fix commit message naming the specific GB10/SM121 constraint it addresses. The cross-fork PR into jasl’s #41834 and the field report on vllm-project/vllm#41063 are both open. A smaller goal on the side is a PR adding a no-container venv path to the DSpark recipe repository this whole thing started from.

The bigger piece of unfinished work is making NVFP4 experts genuinely native instead of falling back through Marlin MXFP4 the way the current port does. That’s blocked upstream on DeepGEMM #372, no arch-12 expert-scale packing, and the DeepGEMM maintainers don’t have SM120 hardware of their own to test a fix against. Two GB10 nodes and a serving stack that already knows how to reproduce its own corner cases is exactly the validation capacity that issue is missing.

Five of the thirteen findings were loud: crashes and hangs during bring-up, layer 1 in the diagram above. Eight were quiet: three bookkeeping bugs that only wasted time, plus five correctness bugs that produced fluent, plausible, wrong output and needed concurrency and uptime just to show up at all. My first two explanations for the quiet bugs, a precision regression in one KV dtype and a numerics failure in the indexer, were each wrong or incomplete once I actually measured them against a controlled A/B. Getting a new model architecture correct on hardware it was never validated against turned out to mean thirteen bugs, not one, and each one only became visible once I had a test that could actually see it.

Posted on

Fixing tensor parallelism for DiffusionGemma in vLLM

The vLLM project logo

Back in June, while experimenting with Diffusion Gemma 4 on vLLM, I've noticed it won't start on my 4 Nvidia RTX A4000 graphic cards. I sent a patch to vLLM that makes DiffusionGemma servable on more than one GPU. It went in as PR #46177 against issue #45719, merged on June 26, and shipped in the v0.25.0 release on July 11. This post covers what was actually broken and why the fix ended up shaped the way it is.

The symptom

DiffusionGemma is a block diffusion model built on the Gemma 4 backbone: 25.2B total parameters, 3.8B active, with a mixture-of-experts FFN that runs 8 of 128 experts per token. At that size most people need several GPUs, which means tensor parallelism. Every launch with --tensor-parallel-size above 1 died during engine warmup:

RuntimeError: a and b must have same reduction dim ... [65536, 2816]

A single GPU was completely fine. That combination, working on one card and crashing on four, is the whole clue.

Where the shapes stop agreeing

The sampler uses self-conditioning: at each denoising step it feeds the model a summary of its own previous prediction. For a discrete model the natural summary of a predicted distribution is the expected token embedding, which is just the distribution multiplied by the embedding matrix.

\mathbf{e}^{sc}_i = \mathbb{E}_{x\sim\mathbf{p}_i}\big[\mathrm{emb}(x)\big] = \sum_{v=1}^{V} p_{i,v}\,\mathbf{W}_{v,:} = \mathbf{p}_i\mathbf{W}

Here \mathbf{p}_i is a distribution over the full vocabulary, which for this model is V = 262{,}144 tokens, and \mathbf{W}\in\mathbb{R}^{V\times d} is the token embedding matrix with hidden size d = 2{,}816. The original code did exactly that:

soft_embeds = torch.matmul(
    probs.to(embed_weight.dtype), embed_weight) * normalizer

The problem is that vLLM shards embeddings along the vocabulary axis. With T tensor-parallel ranks, rank r owns only the contiguous slice [s_r, e_r) of the vocabulary:

\mathbf{W}^{(r)} = \mathbf{W}[s_r{:}e_r,\,:] \;\in\; \mathbb{R}^{(V/T)\times d}

So at T = 4 each rank holds 65,536 rows instead of 262,144. Meanwhile probs is still full width, because the sampler runs on gathered logits. The contraction dimension is V on the left and V/T on the right, so PyTorch refuses.

At T = 1 the shard is the whole matrix, the two dimensions agree, and nothing looks wrong. The bug is only reachable in the configuration a 26B model actually needs.

Diagram showing a full-vocabulary probability tensor and a vocab-sharded embedding matrix meeting in a matmul, producing a dimension mismatch and a warmup crash when TP is greater than 1
A full-vocabulary distribution meets a vocab-sharded embedding. The mismatch only exists when T is greater than 1.

The fix

A matrix product whose contracted axis is partitioned is a sum of partial products over the blocks of that partition. Since the vocabulary partition is disjoint and exhaustive:

\mathbf{P}\mathbf{W} \;=\; \sum_{v=1}^{V}\mathbf{P}_{:,v}\,\mathbf{W}_{v,:} \;=\; \sum_{r=0}^{T-1}\mathbf{P}_{:,\,s_r:e_r}\,\mathbf{W}^{(r)}

Every rank already holds its own \mathbf{W}^{(r)}, and it can slice the matching columns out of probs by itself. So each rank computes a local partial \mathbf{S}^{(r)} of shape [L_c \times d], and one all-reduce with summation puts the exact result on every rank:

\mathbf{E}^{sc} \;=\; \gamma \sum_{r=0}^{T-1}\mathbf{S}^{(r)}, \qquad \mathbf{S}^{(r)} = \mathbf{P}_{:,\,s_r:e_r}\,\mathbf{W}^{(r)}

where \gamma is the learned normalizer. No weight ever moves.

Dataflow diagram: each tensor-parallel rank multiplies its own probability slice by its own embedding shard, and a single all-reduce sums the hidden-sized partials into an identical result on every rank
Each rank computes a partial over its own vocabulary shard. One all-reduce of a small tensor reassembles the exact result everywhere.
local_probs = probs[..., sc_vocab_start:sc_vocab_end] \
    .to(embed_weight.dtype)
soft_embeds = torch.matmul(
    local_probs,
    embed_weight[: sc_vocab_end - sc_vocab_start])
if tp_size > 1:
    soft_embeds = torch.ops.vllm.all_reduce(
        soft_embeds, group_name=tp_group_name)
soft_embeds = soft_embeds * normalizer

This is the same communication pattern as a row-parallel linear layer in Megatron-LM: operands sharded on the contracted axis, local products giving partial sums, one all-reduce reconstructing the output. The only departure from bit-identity with the single GPU result is the order of additions, which is a benign floating point reassociation.

Two details matter here because the sampler is compiled.

The collective is torch.ops.vllm.all_reduce and not the eager tensor_model_parallel_all_reduce. The sampler step is wrapped in @torch.compile and captured into CUDA graphs, and only the functionalized custom op is traceable and graph safe. That constraint comes from the engine rather than from the model, and it decides what the fix is allowed to look like.

The tp_size > 1 guard is also load bearing. At T = 1 the slice is the whole vocabulary, the partial is already the full product, and the collective is skipped, so the single GPU path stays bitwise identical to what it was. The new code generalizes the old code instead of adding a second path that has to be kept in sync with it.

One smaller point worth mentioning: the shard bounds come from the embedding's own metadata, org_vocab_start_index and org_vocab_end_index, rather than from arithmetic on V and T. vLLM pads shard layouts to hardware friendly multiples, and those padding rows carry no probability mass, so reading the true bounds keeps them out of the slice.

Why not just all-gather the weights

There was a simpler fix available. All-gather \mathbf{W} to every rank once at startup, then leave the original matmul alone. It is equally correct. I did not do it because of what it costs.

At V = 262{,}144, d = 2{,}816 and BF16, the embedding is

V d b = 262{,}144 \times 2{,}816 \times 2 = 1{,}476{,}395{,}008 \text{ bytes} \approx 1.38 \text{ GiB}

All-gather puts that on every GPU and keeps it there. The hardware I was testing on is four RTX A4000s at 16 GiB each, already sitting near 15.4 GiB per card in service. An extra 1.38 GiB per rank is the difference between serving and an OOM. The reason anyone reaches for tensor parallelism in the first place is that the model does not fit on one card, so memory pressure is the normal case here, not an edge case.

All-gather also makes every rank redundantly compute the full matmul: 378 GFLOP per rank per step at T = 4, against 94.5 GFLOP for the sharded version.

What the all-reduce design costs instead is a collective on the hot path, one per denoising step. But the tensor being reduced is [L_c \times d], with no factor of V in it anywhere. At canvas length L_c = 256 that is 1.38 MiB, roughly 2.06 MiB per rank per request per step through a ring all-reduce.

Comparison of two correct designs: all-gathering the embedding weight adds 1.38 GiB per rank persistently, while all-reducing the local partials adds zero extra memory
Replicate the weight, or reduce the small activation. Both are correct and the cost profiles are not close.

One caveat on that comparison. If you count only wire volume, the one time all-gather wins over a long enough run. With K denoising steps and N active sequences, the crossover sits at

2 K N L_c d \;\lesssim\; V d \quad\Longleftrightarrow\quad K N L_c \;\lesssim\; V/2

which at these sizes means K N \lesssim 512. The T = 4 runs came in near 11 denoising steps per canvas, well under that for small batches. So the accurate description of the merged design is memory optimal inside vLLM's existing vocab-sharded layout, and communication aware rather than unconditionally communication optimal.

Where it sits in the sampler

Flowchart of one block-diffusion denoising step, from gathered logits through sampling, confidence testing and token commitment, with the self-conditioning soft embedding projection marked with a star
One denoising step. The starred node is the operator that changed.

The starred node is the only place in the loop where a full-vocabulary distribution meets the vocab-sharded embedding, which makes it the only place TP correctness for self-conditioning has to be enforced.

Running it

  • Model: diffusiongemma-26B-A4B-it, INT8 dynamic
  • Hardware: 4 × RTX A4000, 16 GiB each, Ampere SM86, about 15.4 GiB used per GPU in service
  • Config: --tensor-parallel-size 4, TRITON_ATTN, canvas length 256, max model length 131,072
  • Before: warmup crash on the reduction dim
  • After: warmup completes, CUDA graphs capture, four TP workers active, coherent generation

For a throughput sample I asked the service to write a complete offline pixel art editor as a single HTML file. 278 prompt tokens in, 8,814 completion tokens out, 50.56 seconds wall clock, 174.3 tokens/s measured client side. Server side logs showed generation windows up to 230.4 tokens/s. The output ran to 27,005 characters over 793 lines and was a complete, well-formed document.

That rate moves around quite a bit, and it should. Block diffusion commits a variable number of tokens after a variable number of denoising iterations. In that run the logs ranged from 13.3 to 24.2 steps per canvas, and the windows with fewer steps and more tokens committed per step read faster. That variation comes from the sampler's acceptance dynamics rather than from anything in this change.

None of this is a benchmark. There was no T > 1 baseline to be faster than, because T > 1 did not run at all. The claim here is availability, not speedup.

Keeping it fixed

The merged change adds a T = 2 GSM8K evaluation config on the FP8 checkpoint, 1,319 questions, 5-shot. The sharded sampler path now gets exercised on real hardware as part of vLLM's regression surface, rather than only by a synthetic shape test.

What it does not fix

Pipeline parallelism. The original issue reported a PP failure too, but PP needs diffusion canvas state propagated across pipeline stages, which is a different problem from making one projection layout aware. I scoped the change to TP and said so in the PR.

The general point

Autoregressive sampling propagates a sampled token id, so the full vocabulary distribution is a temporary that dies inside the sampler. Diffusion sampling propagates the distribution itself. Any operator that consumes a full-vocabulary distribution against a vocab-sharded parameter therefore has to be rewritten as a sharded reduction, and self-conditioning is simply the first one you hit in DiffusionGemma. For this class of model I doubt it will be the last.

The full derivation, the proof that the sharded version is exact, and the cost model behind that comparison are written up in a pre-print with my doctoral advisor, Sicong Shao.

Posted on

Image Policy Webhooks on Kubernetes (image scanner admission controller)

Adding Trivy Scanner as custom Admission Controller

We will include an Image Policy Webhook on our kubeadm Kubernetes cluster in order to enhance its security, not allowing containers with more than 3 CRITICAL vulnerabilities from getting scheduled on our cluster.

To accomplish this, the first step involves deploying a Scanner. In this instance, I have utilized a custom trivy scanner that I developed in Go, which utilizes the Trivy scanner in its operation. You can review the project here: go-trivy-scanner

Changes required to kube-api

Add the option --admission-control-config-file=/etc/kubernetes/admission-control/image-policy-webhook-conf.yaml

Append the plugin ImagePolicyWebhook to the option --enable-admission-plugins

Add the volume

  volumes:
  - hostPath:
      path: /etc/kubernetes/admission-control
      type: DirectoryOrCreate
    name: etc-kubernetes-admission-control

And the volume-mounts to kube-api

    volumeMounts:
    - mountPath: /etc/kubernetes/admission-control
      name: etc-kubernetes-admission-control
      readOnly: true

Configuration files

Proceeding with our custom Image Policy Webhook.

file /etc/kubernetes/admission-control/image-policy-webhook-conf.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: ImagePolicyWebhook
    path: /etc/kubernetes/admission-control/imagepolicyconfig.yaml

We define the Image Policy Config here:

file /etc/kubernetes/admission-control/imagepolicyconfig.yaml

imagePolicy:
  kubeConfigFile: /etc/kubernetes/admission-control/trivy-scanner.kubeconfig
  allowTTL: 50
  denyTTL: 50
  retryBackoff: 500
  defaultAllow: true

And we define the kubeconfig file, this is the minimal supported configuration to make it work:

file /etc/kubernetes/admission-control/trivy-scanner.kubeconfig

apiVersion: v1
kind: Config
clusters:
- cluster:
    server: https://trivy-scanner<my-domain>/scan
  name: okd
users:
- name: admin
  user: {}
preferences: {}
contexts:
- context:
    cluster: okd
    user: admin
  name: admin
current-context: admin

Reviewing our kube-api static pod

After that, on our master node, we will configure the static Pod kube-api, located on /etc/kubernetes/manifests/kube-apiserver.yaml mounting an admission-controller directory, where we will place our config files.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.124.20:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.124.20
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
    - --admission-control-config-file=/etc/kubernetes/admission-control/image-policy-webhook-conf.yaml
    [...]
    volumeMounts:
    [...]
    - mountPath: /etc/kubernetes/admission-control
      name: etc-kubernetes-admission-control
      readOnly: true
    [...]
  volumes:
  [...]
  - hostPath:
      path: /etc/kubernetes/admission-control
      type: DirectoryOrCreate
    name: etc-kubernetes-admission-control

This will restart our kube-api container

we can validate with

crictl ps -a
crictl logs <container>

Test the Image Policy Webhook

Once kube-api is back online, we can try to deploy a faulty pod with lot of vulnerabilities, this should fail:

file faulty-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: imagepolicy-nginx-pod
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
kubectl create -f faulty-pod.yaml
Error from server (Forbidden): error when creating "faulty-pod.yaml": pods "imagepolicy-nginx-pod" is forbidden: image policy webhook backend denied one or more images: More than 3 CRITICAL vulnerabilities, rejected: [nginx:1.14.2]

Posted on

The force of gravity

Back in 2007, this track helped me to come back on track. I was failing on all the things that I had supposed to do by that age, badly.

It took me around seven years of my life to overcome such self inflicted damage (messing around just after becoming an adult is quite serious thing I guess), as the viral video attests, "you f... around, you will find out". Anyway, I've learned my lesson.

The force of gravity will hurt you if you don't pay attention.

Posted on

Deploy an Elasticsearch cluster for Kubernetes (ECK) on Google Compute Platform (GCP on GKE) with Terraform – Part I

This will be a very technical post but I think that is gonna be also quite interesting if you are working with cloud technologies.

Elasticsearch is a pretty nice technology widely used on big data stuff, analysis and so on. However, this tool is heavy and little bit difficult to deploy and maintain on healthy status.

I'm working a lot with Google Compute Platform (GCP) that's why I decided to include this part as well.

First things first

If you don't have a GCP account, is pretty straightforward to get one, even with some free usage, Google will give you 300 dollars to spend on it... by previous registration with your credit card 😉 go ahead and do it: https://console.cloud.google.com

Also download the gcloud CLI: https://cloud.google.com/sdk/docs/install

We will be using the project called GKE Terraform project as you can check below:

Get access to your gcloud project on the CLI and perform the browser steps needed to achieve it:

$ gcloud auth login

Get access to your project:

Let's create an empty VPC to simulate one environment with previous stuff deployed on it, like other instances and so on.

Well, at this point we have the very basic infrastructure to start using Terraform.

Infrastructure as Code, what does that mean?

Terraform is the leading tool to deploy infrastructure on this way, you can define a very complex set of infrastructure with code functions and treating them like objects and variables.

The GKE Terraform project is available here:

https://github.com/calvarado2004/terraform-gke

Please note that the size of the nodes is huge, you can go ahead and delete some of those pools of nodes and customize the CPU's and memory according to your needs and budget, I will do that, of course. You can check here another branch with smaller nodes: https://github.com/calvarado2004/terraform-gke/tree/resize-to-small

ECK can be deployed on a single node, but the minimal enterprise configuration should have:

  • One Kibana node
  • One Coordinator node
  • One Master node
  • Two Data nodes

This deployment is creating a pool of nodes for each type of node, in order to enable the autoresizing on further moments of the infrastructure lifecycle. That could give you an idea of the complexity that you can handle easily with Terraform.

Kubernetes have two internal layers of networking. We will be using the following three CIDRs:

  • 170.35.0.0/24 for our GCP VPC, the most external face.
  • 10.99.240.0/20 for our Kubernetes services.
  • 10.96.0.0/14 for our Kubernetes Pods.

You can install Terraform if you have Ubuntu using this way:

$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
$ sudo apt-get update && sudo apt-get install terraform

Otherwise, check how to install it on your machine:

https://www.terraform.io/downloads.html

This the content of the file gke.tf

variable "gke_username" {
  default     = ""
  description = "gke username"
}

variable "gke_password" {
  default     = ""
  description = "gke password"
}

variable "cluster_name" {
  default = "gke-cluster"
  description = "cluster name"
}

variable "zone" {
  default = "us-east1-b"
  description = "cluster zone"
}

#Your pods will have an IP address from this CIDR
variable "cluster_ipv4_cidr" {
  default = "10.96.0.0/14"
  description = "internal cidr for pods"
}

#Your Kubernetes services will have an IP from this range
variable "services_ipv4_cidr_block" {
  default = "10.99.240.0/20"
  description = "nternal range for the kubernetes services"
}

# GKE cluster
resource "google_container_cluster" "primary" {
  name     = var.cluster_name
  location = var.zone

  remove_default_node_pool = true
  initial_node_count       = 1

  network                  = google_compute_network.vpc-gke.name
  subnetwork               = google_compute_subnetwork.subnet.name
  cluster_ipv4_cidr        = var.cluster_ipv4_cidr
  services_ipv4_cidr_block = var.services_ipv4_cidr_block

  min_master_version = "1.17.13-gke.2001"	

  master_auth {
    username = var.gke_username
    password = var.gke_password

    client_certificate_config {
      issue_client_certificate = false
    }
  }

  cluster_autoscaling {
    enabled = false
  }

}

# Separately Managed Master Pool
resource "google_container_node_pool" "master-pool" {
  name       = "master-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 2
  }

  management {
    auto_repair  = true
    auto_upgrade = false
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/devstorage.read_only",
    ]

    labels = {
      es_type = "master_nodes"
    }
    # 6 CPUs, 12GB of RAM
    preemptible  = false
    image_type   = "ubuntu_containerd"
    machine_type = "custom-6-12288"
    local_ssd_count = 0
    disk_size_gb    = 50
    disk_type       = "pd-standard"
    tags         = ["gke-node", "${var.cluster_name}-master"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

# Separately Managed Data Pool
resource "google_container_node_pool" "data-pool" {
  name       = "data-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = 2

  autoscaling {
    min_node_count = 2
    max_node_count = 4
  }

  management {
    auto_repair = true
    auto_upgrade = false
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/devstorage.read_only",
    ]

    labels = {
      es_type = "data_nodes"
    }

    # 14 CPUs, 41GB of RAM
    preemptible  = false
    image_type   = "ubuntu_containerd"
    machine_type = "custom-14-41984"
    local_ssd_count = 0
    disk_size_gb    = 50
    disk_type       = "pd-standard"

    tags         = ["gke-node", "${var.cluster_name}-data"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

# Separately Managed Coordinator Pool
resource "google_container_node_pool" "coord-pool" {
  name       = "coord-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 2
  }

  management {
    auto_repair  = true
    auto_upgrade = false
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/devstorage.read_only",
    ]

    labels = {
      es_type = "coordinator_nodes"
    }

    # 6 CPUs, 22GB of RAM
    preemptible  = false
    image_type   = "ubuntu_containerd"
    machine_type = "custom-6-22528"
    local_ssd_count = 0
    disk_size_gb    = 50
    disk_type       = "pd-standard"
    tags         = ["gke-node", "${var.cluster_name}-coord"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

# Separately Managed Kibana Pool
resource "google_container_node_pool" "kibana-pool" {
  name       = "kibana-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 2
  }

  management {
    auto_repair  = true
    auto_upgrade = false
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/devstorage.read_only",
    ]

    labels = {
      es_type = "kibana_nodes"
    }

    # 4 CPUs, 13GB of RAM
    preemptible  = false
    image_type   = "ubuntu_containerd"
    machine_type = "custom-4-13312"
    local_ssd_count = 0
    disk_size_gb    = 50
    disk_type       = "pd-standard"
    tags         = ["gke-node", "${var.cluster_name}-kibana"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

output "kubernetes_cluster_name" {
  value       = google_container_cluster.primary.name
  description = "GKE Cluster Name"
}

And the content of the file vpc.tf

variable "project_id" {
  description = "project id"
}

variable "region" {
  description = "region"
}

provider "google" {
  project = var.project_id
  region  = var.region
}

# VPC
resource "google_compute_network" "vpc-gke" {
  name                    = "${var.cluster_name}-vpc"
  auto_create_subnetworks = "false"
}

# Subnet
resource "google_compute_subnetwork" "subnet" {
  name          = "${var.cluster_name}-subnet"
  region        = var.region
  network       = google_compute_network.vpc-gke.name
  ip_cidr_range = "170.35.0.0/24"

}

#Peering between OLD VMs vpc and GKE K8s vpc
resource "google_compute_network_peering" "to-vms-vpc" {
  name         = "to-vms-vpc-vpc-network"
  network      = google_compute_network.vpc-gke.id
  peer_network = "projects/sigma-scheduler-297405/global/networks/vms-vpc-network"
}

resource "google_compute_network_peering" "to-gke-cluster" {
  name         = "to-gke-cluster-vpc-network"
  network      = "projects/sigma-scheduler-297405/global/networks/vms-vpc-network"
  peer_network = google_compute_network.vpc-gke.id
}

output "region" {
  value       = var.region
  description = "region"
}

#Enable communication from GKE pods to external instances, networks and services outside the Cluster.
resource "google_compute_firewall" "gke-cluster-to-all-vms-on-network" {
  name    = "gke-cluster-k8s-to-all-vms-on-network"
  network = google_compute_network.vpc-portal.id

  allow {
    protocol = "tcp"
  }

  allow {
    protocol = "udp"
  }

  allow {
    protocol = "icmp"
  }

  allow {
    protocol = "esp"
  }

  allow {
    protocol = "ah"
  }

  allow {
    protocol = "sctp"
  }

  source_ranges = ["10.96.0.0/14"]
}

Let's deploy this GKE Cluster with Terraform!

Deploy a whole cluster is quite easy:

$ git clone https://github.com/calvarado2004/terraform-gke.git

$ git checkout resize-to-small
Switched to branch 'resize-to-small'
Your branch is up to date with 'origin/resize-to-small'.

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/google...
- Installing hashicorp/google v3.49.0...
- Installed hashicorp/google v3.49.0 (signed by HashiCorp)

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/google: version = "~> 3.49.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform plan -out=gke-cluster.plan

$ terraform apply "gke-cluster.plan"

Posted on

Deploy a GKE Cluster with Portworx

Let's deploy a nice GKE Cluster with a customized Portworx deployment using security capabilities and encrypted volumes

Get your own GCP account, download gcloud and authenticate on your laptop.

gcloud container clusters create carlos-lab01 \
    --zone us-east1-b \
    --disk-type=pd-ssd \
    --disk-size=50GB \
    --labels=portworx=gke \
    --machine-type=n1-highcpu-8 \
    --num-nodes=5 \
    --image-type ubuntu \
    --scopes compute-rw,storage-ro,cloud-platform \
    --enable-autoscaling --max-nodes=5 --min-nodes=5
    

gcloud container clusters get-credentials carlos-lab01 --zone us-east1-b --project <your-project>

gcloud services enable compute.googleapis.com

Wail until having your cluster available

Then you can install Portworx using the operator.

operator.yaml

# SOURCE: https://install.portworx.com/?comp=pxoperator
apiVersion: v1
kind: ServiceAccount
metadata:
  name: portworx-operator
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
   name: portworx-operator
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: portworx-operator
subjects:
- kind: ServiceAccount
  name: portworx-operator
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: portworx-operator
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: portworx-operator
  namespace: kube-system
spec:
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  replicas: 1
  selector:
    matchLabels:
      name: portworx-operator
  template:
    metadata:
      labels:
        name: portworx-operator
    spec:
      containers:
      - name: portworx-operator
        imagePullPolicy: Always
        image: portworx/px-operator:1.5.0
        command:
        - /operator
        - --verbose
        - --driver=portworx
        - --leader-elect=true
        env:
        - name: OPERATOR_NAME
          value: portworx-operator
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "name"
                    operator: In
                    values:
                    - portworx-operator
              topologyKey: "kubernetes.io/hostname"
      serviceAccountName: portworx-operator

px-enterprisecluster.yaml

# SOURCE: https://install.portworx.com/?operator=true&mc=false&kbver=1.20.8&b=true&kd=type%3Dpd-standard%2Csize%3D150&csicd=true&mz=5&s=%22type%3Dpd-ssd%2Csize%3D150%22&j=auto&c=px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74&gke=true&stork=true&csi=true&mon=true&st=k8s&promop=true
kind: StorageCluster
apiVersion: core.libopenstorage.org/v1
metadata:
  name: px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74
  namespace: kube-system
  annotations:
    portworx.io/install-source: "https://install.portworx.com/?operator=true&mc=false&kbver=1.20.8&b=true&kd=type%3Dpd-standard%2Csize%3D150&csicd=true&mz=5&s=%22type%3Dpd-ssd%2Csize%3D150%22&j=auto&c=px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74&gke=true&stork=true&csi=true&mon=true&st=k8s&promop=true"
    portworx.io/is-gke: "true"
spec:
  image: portworx/oci-monitor:2.8.0
  imagePullPolicy: Always
  kvdb:
    internal: true
  cloudStorage:
    deviceSpecs:
    - type=pd-ssd,size=200
    journalDeviceSpec: auto
    kvdbDeviceSpec: type=pd-standard,size=50
    maxStorageNodesPerZone: 5
  secretsProvider: k8s
  stork:
    enabled: true
    args:
      webhook-controller: "false"
  autopilot:
    enabled: true
    providers:
    - name: default
      type: prometheus
      params:
        url: http://prometheus:9090
  monitoring:
    telemetry:
      enabled: true
    prometheus:
      enabled: true
      exportMetrics: true
  featureGates:
    CSI: "true"
kubectl create clusterrolebinding myname-cluster-admin-binding \
    --clusterrole=cluster-admin --user=`gcloud info --format='value(config.account)'`

kubectl apply -f operator.yaml

kubectl apply -f px-enterprisecluster.yaml
kubectl get all -n kube-system                                                                                                
NAME                                                           READY   STATUS    RESTARTS   AGE
pod/autopilot-7b4f7f58f4-kchs4                                 1/1     Running   0          34m
pod/event-exporter-gke-67986489c8-prn9p                        2/2     Running   0          41m
pod/fluentbit-gke-bn5nm                                        2/2     Running   0          41m
pod/fluentbit-gke-f7k2j                                        2/2     Running   0          41m
pod/fluentbit-gke-h672g                                        2/2     Running   0          41m
pod/fluentbit-gke-n9664                                        2/2     Running   0          41m
pod/fluentbit-gke-xjttt                                        2/2     Running   0          41m
pod/gke-metrics-agent-d64hw                                    1/1     Running   0          41m
pod/gke-metrics-agent-fhw8l                                    1/1     Running   0          41m
pod/gke-metrics-agent-gsfvk                                    1/1     Running   0          41m
pod/gke-metrics-agent-mqm64                                    1/1     Running   0          41m
pod/gke-metrics-agent-wwjvx                                    1/1     Running   0          41m
pod/kube-dns-6c7b8dc9f9-q8v75                                  4/4     Running   0          41m
pod/kube-dns-6c7b8dc9f9-wqthz                                  4/4     Running   0          41m
pod/kube-dns-autoscaler-844c9d9448-4fx8f                       1/1     Running   0          41m
pod/kube-proxy-gke-carlos-lab01-default-pool-a6362dc8-11k6     1/1     Running   0          41m
pod/kube-proxy-gke-carlos-lab01-default-pool-a6362dc8-5lgd     1/1     Running   0          16m
pod/kube-proxy-gke-carlos-lab01-default-pool-a6362dc8-b73f     1/1     Running   0          41m
pod/kube-proxy-gke-carlos-lab01-default-pool-a6362dc8-n5fl     1/1     Running   0          41m
pod/kube-proxy-gke-carlos-lab01-default-pool-a6362dc8-v02w     1/1     Running   0          41m
pod/l7-default-backend-56cb9644f6-xfd65                        1/1     Running   0          41m
pod/metrics-server-v0.3.6-9c5bbf784-9z6sm                      2/2     Running   0          40m
pod/pdcsi-node-4wprs                                           2/2     Running   0          41m
pod/pdcsi-node-685ht                                           2/2     Running   0          41m
pod/pdcsi-node-g42tb                                           2/2     Running   0          41m
pod/pdcsi-node-ln4tw                                           2/2     Running   0          41m
pod/pdcsi-node-ncqhl                                           2/2     Running   0          41m
pod/portworx-api-76kcn                                         1/1     Running   0          34m
pod/portworx-api-887bl                                         1/1     Running   0          34m
pod/portworx-api-br4f2                                         1/1     Running   0          34m
pod/portworx-api-hzfsn                                         1/1     Running   0          34m
pod/portworx-api-zcd4m                                         1/1     Running   0          34m
pod/portworx-kvdb-8ls5k                                        1/1     Running   0          71s
pod/portworx-kvdb-c797z                                        1/1     Running   0          13m
pod/portworx-kvdb-gmxpv                                        1/1     Running   0          13m
pod/portworx-operator-bfc87df78-schcz                          1/1     Running   0          36m
pod/portworx-pvc-controller-696959f9bc-4kj5v                   1/1     Running   0          34m
pod/portworx-pvc-controller-696959f9bc-gn2tw                   1/1     Running   0          34m
pod/portworx-pvc-controller-696959f9bc-jmsxn                   1/1     Running   0          34m
pod/prometheus-px-prometheus-0                                 3/3     Running   1          33m
pod/px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74-9pvws      3/3     Running   0          74s
pod/px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74-hnzs8      3/3     Running   0          88s
pod/px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74-j7vrc      3/3     Running   1          34m
pod/px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74-sgm67      3/3     Running   0          113s
pod/px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74-xvzxn      3/3     Running   0          13m
pod/px-csi-ext-5686675c58-5qfzq                                3/3     Running   0          34m
pod/px-csi-ext-5686675c58-dbmmb                                3/3     Running   0          34m
pod/px-csi-ext-5686675c58-vss9p                                3/3     Running   0          34m
pod/px-prometheus-operator-8c88487bc-jv9fd                     1/1     Running   0          34m
pod/stackdriver-metadata-agent-cluster-level-9548fb7d6-vm552   2/2     Running   0          41m
pod/stork-75dd8b896-g4qqj                                      1/1     Running   0          34m
pod/stork-75dd8b896-mb2xt                                      1/1     Running   0          34m
pod/stork-75dd8b896-zjlwm                                      1/1     Running   0          34m
pod/stork-scheduler-574757dd8d-866bv                           1/1     Running   0          34m
pod/stork-scheduler-574757dd8d-jhx7w                           1/1     Running   0          34m
pod/stork-scheduler-574757dd8d-mhg99                           1/1     Running   0          34m

NAME                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                               AGE
service/default-http-backend        NodePort    10.3.241.138   <none>        80:31243/TCP                          41m
service/kube-dns                    ClusterIP   10.3.240.10    <none>        53/UDP,53/TCP                         41m
service/kubelet                     ClusterIP   None           <none>        10250/TCP                             33m
service/metrics-server              ClusterIP   10.3.248.53    <none>        443/TCP                               41m
service/portworx-api                ClusterIP   10.3.242.28    <none>        9001/TCP,9020/TCP,9021/TCP            34m
service/portworx-operator-metrics   ClusterIP   10.3.247.121   <none>        8999/TCP                              35m
service/portworx-service            ClusterIP   10.3.245.123   <none>        9001/TCP,9019/TCP,9020/TCP,9021/TCP   34m
service/prometheus-operated         ClusterIP   None           <none>        9090/TCP                              33m
service/px-csi-service              ClusterIP   None           <none>        <none>                                34m
service/px-prometheus               ClusterIP   10.3.245.244   <none>        9090/TCP                              34m
service/stork-service               ClusterIP   10.3.242.128   <none>        8099/TCP,443/TCP                      34m

NAME                                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                        AGE
daemonset.apps/fluentbit-gke               5         5         5       5            5           kubernetes.io/os=linux                                               41m
daemonset.apps/gke-metrics-agent           5         5         5       5            5           kubernetes.io/os=linux                                               41m
daemonset.apps/gke-metrics-agent-windows   0         0         0       0            0           kubernetes.io/os=windows                                             41m
daemonset.apps/kube-proxy                  0         0         0       0            0           kubernetes.io/os=linux,node.kubernetes.io/kube-proxy-ds-ready=true   41m
daemonset.apps/metadata-proxy-v0.1         0         0         0       0            0           cloud.google.com/metadata-proxy-ready=true,kubernetes.io/os=linux    41m
daemonset.apps/nvidia-gpu-device-plugin    0         0         0       0            0           <none>                                                               41m
daemonset.apps/pdcsi-node                  5         5         5       5            5           kubernetes.io/os=linux                                               41m
daemonset.apps/pdcsi-node-windows          0         0         0       0            0           kubernetes.io/os=windows                                             41m
daemonset.apps/portworx-api                5         5         5       5            5           <none>                                                               34m

NAME                                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/autopilot                                  1/1     1            1           34m
deployment.apps/event-exporter-gke                         1/1     1            1           41m
deployment.apps/kube-dns                                   2/2     2            2           41m
deployment.apps/kube-dns-autoscaler                        1/1     1            1           41m
deployment.apps/l7-default-backend                         1/1     1            1           41m
deployment.apps/metrics-server-v0.3.6                      1/1     1            1           41m
deployment.apps/portworx-operator                          1/1     1            1           36m
deployment.apps/portworx-pvc-controller                    3/3     3            3           34m
deployment.apps/px-csi-ext                                 3/3     3            3           34m
deployment.apps/px-prometheus-operator                     1/1     1            1           34m
deployment.apps/stackdriver-metadata-agent-cluster-level   1/1     1            1           41m
deployment.apps/stork                                      3/3     3            3           34m
deployment.apps/stork-scheduler                            3/3     3            3           34m

NAME                                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/autopilot-7b4f7f58f4                                  1         1         1       34m
replicaset.apps/event-exporter-gke-67986489c8                         1         1         1       41m
replicaset.apps/kube-dns-6c7b8dc9f9                                   2         2         2       41m
replicaset.apps/kube-dns-autoscaler-844c9d9448                        1         1         1       41m
replicaset.apps/l7-default-backend-56cb9644f6                         1         1         1       41m
replicaset.apps/metrics-server-v0.3.6-57bc866888                      0         0         0       41m
replicaset.apps/metrics-server-v0.3.6-886d66856                       0         0         0       41m
replicaset.apps/metrics-server-v0.3.6-9c5bbf784                       1         1         1       40m
replicaset.apps/portworx-operator-bfc87df78                           1         1         1       36m
replicaset.apps/portworx-pvc-controller-696959f9bc                    3         3         3       34m
replicaset.apps/px-csi-ext-5686675c58                                 3         3         3       34m
replicaset.apps/px-prometheus-operator-8c88487bc                      1         1         1       34m
replicaset.apps/stackdriver-metadata-agent-cluster-level-546484c84b   0         0         0       41m
replicaset.apps/stackdriver-metadata-agent-cluster-level-9548fb7d6    1         1         1       41m
replicaset.apps/stork-75dd8b896                                       3         3         3       34m
replicaset.apps/stork-scheduler-574757dd8d                            3         3         3       34m

NAME                                        READY   AGE
statefulset.apps/prometheus-px-prometheus   1/1     33m

You can try to test the features of Portworx deploying one Statefulset application

kubectl get sc
NAME                             PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
premium-rwo                      pd.csi.storage.gke.io           Delete          WaitForFirstConsumer   true                   29h
px-db                            kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-db-cloud-snapshot             kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-db-cloud-snapshot-encrypted   kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-db-encrypted                  kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-db-local-snapshot             kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-db-local-snapshot-encrypted   kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-replicated                    kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-replicated-encrypted          kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-secure-sc                     kubernetes.io/portworx-volume   Delete          Immediate              false                  28h
standard (default)               kubernetes.io/gce-pd            Delete          Immediate              true                   29h
standard-rwo                     pd.csi.storage.gke.io           Delete          WaitForFirstConsumer   true                   29h
stork-snapshot-sc                stork-snapshot                  Delete          Immediate              true                   37m

Alright then, we need to create a Cluster Wide secret key to handle our encrypted StorageClasses

YOUR_SECRET_KEY=this-is-gonna-be-your-secret-key

kubectl -n kube-system create secret generic px-vol-encryption \
  --from-literal=cluster-wide-secret-key=$YOUR_SECRET_KEY

And apply this secret to Portworx

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl secrets set-cluster-key \
  --secret cluster-wide-secret-key

Once having your cluster wide secret in place, you can enable the cluster security on your storagecluster object, you can achieve this by editing the storagecluster object:

kubectl edit storagecluster -n kube-system

...
spec:
  security:
    enabled: true

And wait for the PX pods to be redeployed. To get access into your PX Cluster after this, you have to get the tokens on your pods.

PORTWORX_ADMIN_TOKEN=$(kubectl -n kube-system get secret px-admin-token -o json \
    | jq -r '.data."auth-token"' \
    | base64 -d)
    
PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl context create admin --token=$PORTWORX_ADMIN_TOKEN    

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[1].metadata.name}')
kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl context create admin --token=$PORTWORX_ADMIN_TOKEN    

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[2].metadata.name}')
kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl context create admin --token=$PORTWORX_ADMIN_TOKEN 


kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl secrets k8s login

Test the cluster with a StatefulSet 

kubectl create namespace cassandra

Label three of your nodes with the label app=cassandra because this StatefulSet uses this label as node affinity policy.

kubectl label nodes <node01> <node02> <node03> app=cassandra

cassandra.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  namespace: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: app
                operator: In
                values:
                - cassandra
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - cassandra
            topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: cassandra:3.11
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.cassandra.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          tcpSocket:
            port: 9042
          initialDelaySeconds: 30
          timeoutSeconds: 7
        volumeMounts:
        - name: cassandra-data
          mountPath: /var/lib/cassandra
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: px-db-encrypted
      resources:
        requests:
          storage: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: cassandra
  namespace: cassandra
spec:
  clusterIP: None
  selector:
    app: cassandra
  ports:
    - protocol: TCP
      name: port9042k8s
      port: 9042
      targetPort: 9042

Apply this file

kubectl apply -f cassandra.yaml
kubectl get pvc -n cassandra                                
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
cassandra-data-cassandra-0   Bound    pvc-81e11ede-e78a-4fd5-ae64-1ca451d8c8f9   2Gi        RWO            px-db-encrypted   116m
cassandra-data-cassandra-1   Bound    pvc-a83b23ee-1426-4b78-ae29-f6a562701e68   2Gi        RWO            px-db-encrypted   113m
cassandra-data-cassandra-2   Bound    pvc-326a2419-cf50-41d3-93d0-63dbecffbcdd   2Gi        RWO            px-db-encrypted   111m
Posted on

Chess and more

I will start from the beginning.

I started with chess back in 2002, when my brother and I used to play at a small chess club, we were playing just for fun primarily, however, it was pretty exciting because there was many adult players with a solid career on their own fields, such as lawyers, physicists, doctors, and so on.

That environment was very formative for me, even more, one of those professionals told me about Linux and that was the first time that I got curious about it...

But anyway, enough of talking about the past, on this post entry, I'd like to show my path to beat these different bots available to play at Chess.com app.

I started beating a bot with 1200 ELO of chess strength, and now I'm dealing with a bot with 2100 ELO, which is pretty difficult to beat, I mean, starting with 2000 ELO is technically a chess master candidate level, which is certainly is not very easy to achieve.

But I just wanna show here the most interesting games that I played against all these bots using the pretty fancy PGN plugin available and also showing the gif version of the matches.

The first game was against the bot Xavier with 1900 ELO, I played the London System:

The second game was against Li, a bot with 2000 ELO, I played the Alapin Variation against the Sicilian Defense:

This was the game agains Fatima, another bot with 2000 ELO:

And the last game that I'd like to show here will be this wild game against Charles, another bot with 2000 ELO, it started with a London System but I switched into a some kind of 150 Attack:

Posted on

CI/CD example with Python, Django, Kubernetes and Okteto

As a sysadmin with experience providing tech support for enterprise applications for around a decade, all this DevOps stuff happened suddenly and little bit silently to be honest, mostly because when your main concerns are to keep the daily operations working properly and the IT infrastructure doing well, there is no much time to look for new and amazing technologies. I arrived little bit late to this wave but in Mexico nowadays (late 2020), many companies are not even understanding what's going on with all this stuff.

But fortunately for me, in 2017, my colleague César Olea told me on a nice conversation that all this DevOps, CI/CD and remarkably, Containers and Kubernetes were being a great success on the IT Industry; so, after that conversation, I started to look into all this new world on my own, first with Docker and after that with Rancher 1.6 directly, which was simply great for me because that version of Rancher was a glorified Docker-Compose application, that I felt like having my own data center working inside my laptop, with each container performing pretty much like a server. And that was pretty important for me, to be able to understand and adopt all this new technology and this new approach.

Sooner than later, I realized that all this containers are not just another fancy way to deal with applications, nope, is much more than that, is a complete set of practices to enhance and improve the entire IT department, even sometimes called as Digital Transformation, term that if your company is really being involved into that, could be fairly appropriate to use.

Everything working together makes a ton of sense

Now in 2020, everything is still moving forward quite fast, even this basic example will become irrelevant in few months, but anyway, this blog is mine and is pretty much an attempt to demonstrate for myself and for others that actually I have all these skills.

The list of technologies that I'm using is just the basic for a minimal CI/CD architecture:

  • Programming IDE, Visual Studio Code with all the relevant plugins installed.
  • Python with Django as programming language, I'm learning right now Python and Django/Flask
  • Github account and a repository to push all my code.
  • A Kubernetes cluster to deploy all my stuff and deploy Jenkins, I choose Okteto because is cheap and is nice.
  • Okteto internal registry for the container images.
  • Okteto ingress controllers to publish the application.
  • Jenkins as Continuous Integration/Continuous Deployment engine, this approach actually could be different on many companies, because many organizations just set a trigger on every new image pushed to the container registry.

Backend

  • PostgreSQL database managed by Django using Django Models.
  • Python using Django.
  • I'm generating an API Rest to be consumed by the Frontend, however, there are some parts that actually comes directly from Django because this framework can act as frontend as well as backend.
  • Nginx Load Balancer, this would not be needed on this very basic application, but I decided to include it just for the challenge and because on more complex applications Nginx is widely used.
  • Repository URL: https://github.com/calvarado2004/django-api-rest
  • Backend URL: https://nginx-api-rest.calvarado04.com/api/element

Frontend

Infrastructure

Kubernetes is the professional way to deploy and use containers on enterprise graded environments, change my mind hahaha.

So, that is the natural step that you must take if you are being involved with containers.

My K8s namespace looks like:

CI/CD Pipelines!

As you should know, Jenkins works with Groovy to build its pipelines, Groovy it is not my favorite language but is still usable and Jenkins have some useful help on the application itself as well as on its documentation.

Database deployment:

#!/usr/bin/env groovy

//Author: Carlos Alvarado
//Jenkins Pipeline to handle the Continuous Integration and Continuous Deployment on Okteto.


node {
    env.OKTETO_DIR = tool name: 'okteto', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.HOME = "${WORKSPACE}"
    env.KUBECTL_DIR = tool name: 'kubectl', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.GIT_PROJECT = 'https://github.com/calvarado2004/django-api-rest.git'
    
    
    stage ('Download the source code from GitHub'){
            git url: "${GIT_PROJECT}"
    }
    
    
    stage('Deploy the PostgreSQL Database'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            cd ${HOME}/db-k8s
            ${OKTETO_DIR}/okteto namespace
            ${KUBECTL_DIR}/kubectl apply -f kubernetes.yaml
            ${KUBECTL_DIR}/kubectl rollout status deployment.apps/django-api-rest-db-deployment
            '''
            println output
        }
    }
}

Backend deployment pipeline, Django:

#!/usr/bin/env groovy

//Author: Carlos Alvarado
//Jenkins Pipeline to handle the Continuous Integration and Continuous Deployment on Okteto.
//Prerequisites: you should install the Custom tools plugin on Jenkins, ... 
//...get the okteto CLI and Kubectl. You also need to get your Okteto Token and save it on a Jenkins Credential


node {
    
    env.OKTETO_DIR = tool name: 'okteto', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.HOME = "${WORKSPACE}"
    env.CONTAINER_IMAGE = 'registry.cloud.okteto.net/calvarado2004/backend-django'
    env.KUBECTL_DIR = tool name: 'kubectl', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.GIT_PROJECT = 'https://github.com/calvarado2004/django-api-rest.git'
    
    stage ('Prepare Environment with Okteto ') {
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            cleanWs deleteDirs: true
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            '''
            println output
        }
    }
    
    stage ('Download the source code from GitHub'){
            git url: "${GIT_PROJECT}"
    }
    
    stage ('Build and Push Image with Okteto'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            ${OKTETO_DIR}/okteto build -t ${CONTAINER_IMAGE}:${BUILD_TAG} .
            '''
            println output
        }
    }
    
    stage('Deploy the new image to okteto'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            cd ${HOME}/backend-k8s
            ${OKTETO_DIR}/okteto namespace
            cat kubernetes.j2 | sed "s#{{ CONTAINER_IMAGE }}:{{ TAG_USED }}#${CONTAINER_IMAGE}:${BUILD_TAG}#g" > kubernetes.yaml
            ${KUBECTL_DIR}/kubectl apply -f kubernetes.yaml
            ${KUBECTL_DIR}/kubectl rollout status deployment.apps/django-api-rest
            '''
            println output
        }
    }
}

Nginx pipeline

#!/usr/bin/env groovy

//Author: Carlos Alvarado
//Jenkins Pipeline to handle the Continuous Integration and Continuous Deployment on Okteto.
//Prerequisites: you should install the Custom tools plugin on Jenkins, ... 
//...get the okteto CLI and Kubectl. You also need to get your Okteto Token and save it on a Jenkins Credential


node {
    
    env.OKTETO_DIR = tool name: 'okteto', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.HOME = "${WORKSPACE}"
    env.CONTAINER_IMAGE = 'registry.cloud.okteto.net/calvarado2004/backend-django'
    env.KUBECTL_DIR = tool name: 'kubectl', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.GIT_PROJECT = 'https://github.com/calvarado2004/django-api-rest.git'
    
    stage ('Prepare Environment with Okteto ') {
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            cleanWs deleteDirs: true
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            '''
            println output
        }
    }
    
    stage ('Download the source code from GitHub'){
            git url: "${GIT_PROJECT}"
    }
    
    
    stage('Deploy Nginx to okteto'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            cd ${HOME}/nginx-k8s
            ${OKTETO_DIR}/okteto namespace
            ${KUBECTL_DIR}/kubectl apply -f kubernetes.yaml
            ${KUBECTL_DIR}/kubectl rollout status deployment.apps/nginx-api-rest
            '''
            println output
        }
    }
}

Vue pipeline:

#!/usr/bin/env groovy

//Author: Carlos Alvarado
//Jenkins Pipeline to handle the Continuous Integration and Continuous Deployment on Okteto.
//Prerequisites: you should install the Custom tools plugin on Jenkins, ... 
//...get the okteto CLI and Kubectl. You also need to get your Okteto Token and save it on a Jenkins Credential


node {
    
    env.OKTETO_DIR = tool name: 'okteto', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.HOME = "${WORKSPACE}"
    env.CONTAINER_IMAGE = 'registry.cloud.okteto.net/calvarado2004/frontend-vue'
    env.KUBECTL_DIR = tool name: 'kubectl', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.GIT_PROJECT = 'https://github.com/calvarado2004/vuedjango.git'
    
    stage ('Prepare Environment with Okteto ') {
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            cleanWs deleteDirs: true
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            '''
            println output
        }
    }
    
    stage ('Download the source code from GitHub'){
            def output = sh returnStdout: true, script: '''git clone "${GIT_PROJECT}"'''
            println output
    }
    
    stage ('Build and Push Image with Okteto'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            cd ${HOME}/vuedjango
            ${OKTETO_DIR}/okteto build -t ${CONTAINER_IMAGE}:${BUILD_TAG} .
            '''
            println output
        }
    }
    
    stage('Deploy the new image to okteto'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            cd ${HOME}/vuedjango/frontend-k8s
            ${OKTETO_DIR}/okteto namespace
            cat kubernetes.j2 | sed "s#{{ CONTAINER_IMAGE }}:{{ TAG_USED }}#${CONTAINER_IMAGE}:${BUILD_TAG}#g" > kubernetes.yaml
            ${KUBECTL_DIR}/kubectl apply -f kubernetes.yaml
            ${KUBECTL_DIR}/kubectl rollout status deployment.apps/django-api-rest
            '''
            println output
        }
    }
}

Dockerfiles

Docker is just a company that works with containers, but its Dockerfiles became the standard way to define almost all of them.

Here is the Dockerfile for the backend:

FROM python:3.8.5

COPY djangovue /djangovue
COPY requirements.txt /djangovue/requirements.txt
WORKDIR /djangovue
RUN pip install -r requirements.txt && chmod 755 /djangovue/manage.py
CMD python manage.py runserver 0.0.0.0:8000

And the frontend one:

# build environment
FROM node:12.2.0-alpine as build
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH
COPY package.json /app/package.json
RUN npm install --silent
RUN npm install @vue/cli@3.7.0 -g
COPY . /app
RUN npm run build

# production environment
FROM nginx:1.16.0-alpine
COPY --from=build /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Working with more than one environment

If you check this frontend, you will see two environment files, concretely a file called .env.development with the following content:

VUE_APP_DJANGO_HOST=localhost
VUE_APP_DJANGO_PORT=8000
VUE_APP_DJANGO_PROTOCOL=http

and a file called .env.production with something more interesting:

VUE_APP_DJANGO_HOST=nginx-api-rest.calvarado04.com
VUE_APP_DJANGO_PORT=443
VUE_APP_DJANGO_PROTOCOL=https

So, yes, this is the proper way to deal with more than one environment, define inside your code variables that you can check and modify later. Nevermore the developers mantra: but it works on my machine!...

Kubernetes definitions

Database:

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: django-api-rest-pvc
  namespace: calvarado2004
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Secret
metadata:
  name: django-api-rest-credentials
  namespace: calvarado2004
type: Opaque
data:
  user: UG9zdGdyZXM=
  password: UG9zdGdyZXNrOHMk 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-api-rest-db-deployment
  namespace: calvarado2004
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django-api-rest-db-container
  template:
    metadata:
      labels:
        app: django-api-rest-db-container
        tier: backend
    spec:
      containers:
        - name: django-api-rest-db-container
          image: postgres:12.4
          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: user

            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: password

            - name: POSTGRES_DB
              value: djangovuedb

            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata

          ports:
            - containerPort: 5432
          volumeMounts:
            - name: django-api-rest-volume-mount
              mountPath: "/var/lib/postgresql/data"

      volumes:
        - name: django-api-rest-volume-mount
          persistentVolumeClaim:
            claimName: django-api-rest-pvc
---
kind: Service
apiVersion: v1
metadata:
  name: django-api-rest-db-service
  namespace: calvarado2004
spec:
  selector:
    app: django-api-rest-db-container
  ports:
    - protocol: TCP
      port: 5432
      targetPort: 5432

Backend:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-api-rest
  namespace: calvarado2004
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django-api-rest-container
  template:
    metadata:
      labels:
        app: django-api-rest-container
    spec:
      containers:
        - name: django-api-rest-container
          image: {{ CONTAINER_IMAGE }}:{{ TAG_USED }}
          ports:
            - containerPort: 8000
          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: user
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: password
            - name: POSTGRES_HOST
              value: django-api-rest-db-service
      initContainers:
        - name: django-api-rest-init
          image: {{ CONTAINER_IMAGE }}:{{ TAG_USED }}
          command: ['python', 'manage.py', 'migrate']
          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: user
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: password
            - name: POSTGRES_HOST
              value: django-api-rest-db-service
---
kind: Service
apiVersion: v1
metadata:
  name: django-api-rest
  namespace: calvarado2004
spec:
  selector:
    app: django-api-rest-container
  type: ClusterIP
  ports:
  - name: django-http
    protocol: TCP
    port: 8000
    targetPort: 8000

Nginx:

Note that I'm consuming the Django application making reference to the internal DNS that is the standard way to do it on Kubernetes {{application}}.{{namespace}}.svc.cluster.local when you want to consume a service with another service internally. This approach will not work for Vue because that application is effectively consuming the API on client side (literally is doing its duty on your browser) and because of that, it needs to be referenced to the API published to Internet (or Intranet if is an internal app).

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config-map
data:
  nginx.conf: |-
    events {
      
    }
    http {
      include /etc/nginx/conf.d/*.conf;
      upstream backend_server {
          server django-api-rest.calvarado2004.svc.cluster.local:8000;
      }
      server {
          listen 80 default_server;
          server_name nginx-api-rest.calvarado04.com;
          location / {
              proxy_pass http://backend_server;
              proxy_set_header Host $http_host;
              proxy_redirect off;
              proxy_set_header X-Real-IP $remote_addr;
              proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
              proxy_set_header X-Forwarded-Proto $https;
              proxy_connect_timeout 360s;
              proxy_read_timeout 360s;
              proxy_hide_header Access-Control-Allow-Origin;
              proxy_hide_header Access-Control-Allow-Credentials;
              set $CORS_CREDS true;
              set $CORS_ORIGIN $http_origin;
              set $CORS_METHODS 'GET, POST, PUT, DELETE, OPTIONS';
              set $CORS_HEADERS 'Authentication-Token, Cache-Control, Cookie, If-Modified-Since, Range, User-Agent, X-Requested-With';
              set $CORS_EXPOSE_HEADERS 'Content-Disposition, Content-Length, Content-Range, Set-Cookie';
              set $CORS_PREFLIGHT_CACHE_AGE 600;
              set $X_FRAME_OPTIONS '';
              if ($request_method = 'OPTIONS') {
                add_header Access-Control-Allow-Origin $CORS_ORIGIN;
                add_header Access-Control-Allow-Methods $CORS_METHODS;
                add_header Access-Control-Allow-Headers $CORS_HEADERS;
                add_header Access-Control-Allow-Credentials $CORS_CREDS;
                add_header Access-Control-Max-Age $CORS_PREFLIGHT_CACHE_AGE;
                add_header Content-Type 'text/plain; charset=utf-8';
                add_header Content-Length 0;
                return 204;
              }
              if ($request_method != 'OPTIONS') {
                add_header Access-Control-Allow-Origin $CORS_ORIGIN;
                add_header Access-Control-Allow-Methods $CORS_METHODS;
                add_header Access-Control-Allow-Headers $CORS_HEADERS;
                add_header Access-Control-Allow-Credentials $CORS_CREDS;
                add_header Access-Control-Expose-Headers $CORS_EXPOSE_HEADERS;
                add_header X-Frame-Options $X_FRAME_OPTIONS;
              }
          }
      }
 
    }  
---      
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-api-rest
  namespace: calvarado2004
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-api-rest-container
  template:
    metadata:
      labels:
        app: nginx-api-rest-container
    spec:
      containers:
        - name: nginx-api-rest-container
          image: nginx:latest
          volumeMounts: 
            - name: nginx-config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
          ports:
            - containerPort: 443
          command: ["/bin/sh"]
          args: ["-c", "while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\""]
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-config-map
---
kind: Service
apiVersion: v1
metadata:
  name: nginx-api-rest
  namespace: calvarado2004
  annotations:
    dev.okteto.com/auto-ingress: "true"
spec:
  selector:
    app: nginx-api-rest-container
  type: ClusterIP
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80

Frontend:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vue-api-rest
  namespace: calvarado2004
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vue-api-rest-container
  template:
    metadata:
      labels:
        app: vue-api-rest-container
    spec:
      containers:
        - name: vue-api-rest-container
          image: {{ CONTAINER_IMAGE }}:{{ TAG_USED }}
          ports:
            - containerPort: 80

---
kind: Service
apiVersion: v1
metadata:
  name: vue-api-rest
  namespace: calvarado2004
  annotations:
    dev.okteto.com/auto-ingress: "true"
spec:
  selector:
    app: vue-api-rest-container
  type: ClusterIP
  ports:
    - name: "vue-api-rest"
      protocol: TCP
      port: 80
      targetPort: 80

Last thoughts

As you can realize, DevOps adoption is not easy at all because implies to understand and know how to make it work together a huge range of technologies, that used to be very specialized and kind of isolated ones from each others. Developers needs to know more in deep about infrastructure, and Sysadmins, DBA's, Testers and Security teams needs to understand and make some effort to achieve a confortable way to deploy easily to production but warranting the best levels of quality at the same time.

This is the deal, but at the end of the day, it's not rocket science... 😉