Overview

A block is the basic building unit of a blockchain, holding a group of verified transactions along with cryptographic references to preceding blocks. Together, they create an unchangeable ledger within the blockchain network.

In essence, each block serves as a digital container that securely and permanently records transaction information for the entire system.

A block in Bitcoin is like a page in a ledger that records and secures recent transactions.

Each block is linked to the previous one, forming the blockchain.

Structure of a Bitcoin Block

A Bitcoin block has two main parts The Block Header and The Block Body

Block Header

The metadata of the block, containing:

• Version → Protocol version used to create the block.
• Previous Block Hash → Cryptographic hash of the previous block, linking the chain.
• Merkle Root → A hash representing all transactions in the block, built via a Merkle Tree.
• Timestamp → Approximate creation time of the block.
• Difficulty Target (nBits) → The target value that the block hash must meet (controls mining difficulty).
• Nonce → A 32-bit number miners vary to find a valid block hash.

Block Body

• Contains the list of transactions included in the block.

• Starts with a coinbase transaction:

  • The first transaction in every block.
  • Rewards the miner with new Bitcoin (block subsidy + transaction fees).

Block Constraints

Block Size & Limits

• Current average block size: ~1–2 MB (due to SegWit optimization).
• Can hold a few thousand transactions depending on size.

Block Reward

• New bitcoins are created via the block reward in the coinbase transaction.
• Halves every 210,000 blocks (~4 years).
• Started at 50 BTC (2009) → now 6.25 BTC (until 2024 halving → 3.125 BTC).

Block Time

• Target: 10 minutes per block.
• Achieved by adjusting difficulty every 2016 blocks (~2 weeks) so the average time stays near 10 minutes.

Block Validation

• A valid block must:

  • Have a hash below the difficulty target (proof-of-work).
  • Contain valid, non-duplicate transactions.
  • Correctly reference the previous block.
  • Correctly compute the Merkle root.

Blockchain Linking

• Each block references the hash of the previous block.

• This makes the blockchain immutable:

  • Changing one block invalidates all subsequent blocks (since the hashes break).

Block Header

At the top of every Bitcoin block is an 80-byte (160 hex character) block header that summarizes all the data in the block. This header is crucial for:

• Block identification and validation
• Creating the block hash
• Maintaining blockchain integrity through cryptographic linking

Miners repeatedly hash this block header to try and get a result below the current target. Successfully finding such a hash allows the block to be added to the blockchain, a process known as mining.

Block Header Structure

The 80-byte header contains six key fields:

Retrieving Block Header Data

To get the block header for block height 1 using Blockchain.com’s API:

curl 'https://blockchain.info/rawblock/1?format=hex' | head -c 160 > block-1.txt

010000006fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d6190000000000982051fd1e4ba744bbbe680e1fee14677ba1a3c3540bf7b1cdb606e857233e0e61bc6649ffff001d01e36299

Decoding the Block Header

Here’s a Python function to decode a block header from hex format:

import struct
import binascii
import datetime

def decode_block_header(hex_header):
    header_bytes = bytes.fromhex(hex_header)
    
    version = struct.unpack('<I', header_bytes[0:4])[0]
    prev_block = header_bytes[4:36][::-1].hex()
    merkle_root = header_bytes[36:68][::-1].hex()
    timestamp = struct.unpack('<I', header_bytes[68:72])[0]
    bits = struct.unpack('<I', header_bytes[72:76])[0]
    nonce = struct.unpack('<I', header_bytes[76:80])[0]

    return {
        'version': version,
        'previous_block_hash': prev_block,
        'merkle_root': merkle_root,
        'timestamp': f"{timestamp} ({datetime.fromtimestamp(timestamp, UTC).strftime('%Y-%m-%d %H:%M:%S %Z')})",
        'bits': bits,
        'nonce': nonce
    }

#-----------------------------------------------------------------------------------------------------------
def main():
    # Example usage
    hex_header = '010000006fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d6190000000000982051fd1e4ba744bbbe680e1fee14677ba1a3c3540bf7b1cdb606e857233e0e61bc6649ffff001d01e36299'
    decoded = decode_block_header(hex_header)

    for key, value in decoded.items():
        print(f"{key}: {value}")

#-----------------------------------------------------------------------------------------------------------
if __name__ == "__main__":
    main()

Example Output

version: 1
previous_block_hash: 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
merkle_root: 0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098
timestamp: 1231469665 (2009-01-09 02:54:25)
bits: 486604799
nonce: 2573394689

Key Notes

• All integer fields are stored in little-endian format
• Hash fields (Previous Block Hash and Merkle Root) need byte reversal for proper display
• The timestamp represents when mining began, not necessarily when the block was found

Block Body

The block body contains the transactions that the block commits to. It immediately follows the 80-byte block header.

The block body (unlike the header) is variable length and encodes all spending activity included by miners in that block.

The block body is where all the actual bitcoin transfers happen - the header just contains metadata and proof-of-work, while the body contains the economic activity.

Block Body Structure

Transaction Serialization (legacy)

[version: 4 bytes LE]
[input_count: varint]
  for each input:
    [prev_txid: 32 bytes (little-endian in storage)]
    [prev_index: 4 bytes LE]
    [scriptSig_length: varint]
    [scriptSig: scriptSig_length bytes]
    [sequence: 4 bytes LE]
[output_count: varint]
  for each output:
    [value: 8 bytes LE]        # satoshis
    [pk_script_length: varint]
    [pk_script: pk_script_length bytes]
[locktime: 4 bytes LE]

Notes

• prev_txid is stored in byte-order used within the block (usually little-endian in raw bytes); when displayed by explorers the bytes are reversed to big-endian hex.
• value is an unsigned 8-byte little-endian integer of satoshis.
• sequence historically used for RBF / timelocks; typically 0xffffffff.

Transaction Serialization (SegWit - BIP141)

[version: 4 bytes LE]
[marker: 1 byte == 0x00]
[flag:   1 byte == 0x01]
[input_count: varint]
  for each input: (same fields as legacy *but* scriptSig is usually small or empty)
[output_count: varint]
  for each output: (same as legacy)
[witnesses for each input]
[locktime: 4 bytes LE]

Notes

• SegWit transactions add a 2-byte header after version
• for each stack element: [witness_item_len: varint] [witness_item: bytes]
• The presence of SegWit is detected when marker == 0x00 and flag != 0x00 immediately after the version.

Retrieving Block Body Data

To get the block body for block height 1 using Blockchain.com’s API:

curl 'https://blockchain.info/rawblock/1?format=hex' | tail -c +161 > block-1-body.txt

0101000000010000000000000000000000000000000000000000000000000000000000000000ffffffff0704ffff001d0104ffffffff0100f2052a0100000043410496b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52da7589379515d4e0a604f8141781e62294721166bf621e73a82cbf2342c858eeac00000000

Decoding the genesis block.

import json
import struct
import requests
from typing import TypedDict


class BlockHeader(TypedDict):
    """Type definition for decoded block header data."""

    version: int
    previous_block_hash: str
    merkle_root: str
    timestamp: int
    bits: int
    nonce: int


class TransactionInput(TypedDict):
    """Type definition for decoded block tx input data."""

    previous_tx_hash: str
    previous_output_index: int
    script_sig: str
    script_sig_length: str
    sequence: str
    is_coinbase: bool


class TransactionOutput(TypedDict):
    """Type definition for decoded block transaction output data."""

    value_satoshi: int
    value_btc: float
    script_pubkey: str
    script_pubkey_length: int


class WitnessItem(TypedDict):
    """Type definition for a single witness item."""
    data: str
    length: int
    type: str
    description: str


class TransactionWitness(TypedDict):
    """Type definition for witness data of a single input."""
    input_index: int
    witness_count: int
    witness_items: list[WitnessItem]
    witness_type: str
    total_witness_size: int


class Transaction(TypedDict):
    version: int
    tx_type: str
    has_witness: bool
    tx_input_count: int
    tx_inputs: list[TransactionInput]
    tx_output_count: int
    tx_outputs: list[TransactionOutput]
    lock_time: int
    size_bytes: int
    total_output_value_satoshi: int
    total_output_value_btc: float
    tx_witnesses: list[TransactionWitness]


class BlockDecoder:
    """
    A comprehensive Bitcoin block decoder that handles both Legacy and SegWit transactions.
    Provides methods to decode block headers, block bodies, and complete blocks.
    """

    def __init__(self):
        """Initialize the Bitcoin block decoder."""
        pass

    # --------------------------------------------------------------------------------------------
    def decode_full_block(self, block):
        """
        """
        try:
            # First 80 bytes (160 hex characters) are the header
            if len(block) < 160:
                raise ValueError("Block data too short to contain a valid header")

            header = block[:160]
            body = block[160:]

            decoded_header = self.decode_block_header(header)
            decoded_body = self.decode_block_body(body)

            return {
                'block_hash': self.calculate_block_hash(header),
                'header': decoded_header,
                'body': decoded_body,
                'total_block_size_bytes': len(block) // 2,
                'header_size_bytes': 80,
                'body_size_bytes': len(body) // 2
            }

        except Exception as ex:
            raise ValueError({
                'error': f"Failed to decode full block: {str(ex)}",
                'raw_hex_length': len(block)
            })

    # --------------------------------------------------------------------------------------------
    @staticmethod
    def decode_block_header(header: str) -> BlockHeader:
        """

        """
        # Validate input
        match header:
            case str() if header.strip():
                header = header.strip().lower()
            case _:
                raise ValueError("Header must be a non-empty string")

        # Validate hex string length (80 bytes = 160 hex characters)
        if len(header) != 160:
            raise ValueError(f"Header must be exactly 160 hex characters, got {len(header)}")

        # Validate hex string format and convert to bytes
        try:
            header_bytes = bytes.fromhex(header)
        except ValueError as e:
            raise ValueError(f"Invalid hex string: {e}") from e

        try:
            (version,) = struct.unpack("<I", header_bytes[0:4])
            previous_block_hash = header_bytes[4:36][::-1].hex()
            merkle_root = header_bytes[36:68][::-1].hex()
            (timestamp,) = struct.unpack("<I", header_bytes[68:72])
            (bits,) = struct.unpack("<I", header_bytes[72:76])
            (nonce,) = struct.unpack("<I", header_bytes[76:80])
        except ValueError as e:
            raise struct.error(f"Failed to unpack header data: {e}") from e

        block_header: BlockHeader = {
            "version": version,
            "previous_block_hash": previous_block_hash,
            "merkle_root": merkle_root,
            "timestamp": timestamp,
            "bits": bits,
            "nonce": nonce,
        }
        return block_header

    # --------------------------------------------------------------------------------------------
    @staticmethod
    def calculate_block_hash(header) -> str:
        """
        Calculate the block hash from the header.

        Args:
            header (str): Block header as hex string

        Returns:
            str: Block hash (double SHA-256, reversed)
        """
        try:
            import hashlib
            header_bytes = bytes.fromhex(header)
            hash1 = hashlib.sha256(header_bytes).digest()
            hash2 = hashlib.sha256(hash1).digest()
            return hash2[::-1].hex()  # Reverse byte order
        except Exception as ex:
            raise ValueError(f"Error calculating hash: {str(ex)}")

    # --------------------------------------------------------------------------------------------
    def decode_block_body(self, body: str):
        """
        """

        # Validate data
        match body:
            case str() if body.strip():
                body = body.strip().lower()
            case _:
                raise ValueError("Block body data must be a non-empty string")

        # Validate hex string format and convert to bytes
        try:
            body_bytes: bytes = bytes.fromhex(body)
        except ValueError as ex:
            raise ValueError(f"Invalid hex string: {ex}") from ex

        try:
            offset: int = 0
            transactions = []
            total_inputs = 0
            total_outputs = 0
            total_value_satoshi = 0
            has_segwit_txs = False
            coinbase_count = 0

            # Transaction count (varint)
            tx_count, offset = self.decode_varint(body_bytes, offset)

            # Extract and retrieve the list of raw hex txs.
            raw_txs: list[str] = self.extract_raw_txs(body_bytes.hex())

            for raw_tx in raw_txs:
                try:
                    tx: Transaction = self.decode_raw_tx(raw_tx)
                    transactions.append(tx)
                    total_inputs += tx['tx_input_count']
                    total_outputs += tx['tx_output_count']
                    total_value_satoshi += tx['total_output_value_satoshi']

                    # Check if any transaction is SegWit
                    if tx['has_witness']:
                        has_segwit_txs = True

                    # Count coinbase transactions
                    if any(inp['is_coinbase'] for inp in tx['tx_inputs']):
                        coinbase_count += 1

                except Exception as ex:
                    raise Exception(f"Error decoding raw_tx {raw_tx}: {ex}") from ex

            return {
                'transaction_count': tx_count,
                'transactions_decoded': len([tx for tx in transactions if 'error' not in tx]),
                'transactions_failed': len([tx for tx in transactions if 'error' in tx]),
                'block_type': 'SegWit Block' if has_segwit_txs else 'Legacy Block',
                'has_segwit_transactions': has_segwit_txs,
                'coinbase_transactions': coinbase_count,
                'total_inputs': total_inputs,
                'total_outputs': total_outputs,
                'total_output_value_satoshis': total_value_satoshi,
                'total_output_value_btc': total_value_satoshi / 100000000.0,
                'block_body_size_bytes': len(body_bytes),
                'transactions': transactions
            }

        except Exception as ex:
            raise Exception(f"Error decoding block body {ex}") from ex

    # -------------------------------------------------------------------------------------------------
    def decode_raw_tx(self, tx: str) -> Transaction:
        """
        """

        # Validate data
        match tx:
            case str() if tx.strip():
                tx = tx.strip().lower()
            case _:
                raise ValueError("Transaction data must be a non-empty string")

        # Validate hex string format and convert to bytes
        try:
            tx_bytes: bytes = bytes.fromhex(tx)
        except ValueError as e:
            raise ValueError(f"Invalid hex string: {e}") from e

        try:
            offset: int = 0

            # Version (4 bytes)
            version: int = struct.unpack('<I', tx_bytes[offset:offset + 4])[0]
            offset += 4

            # Check for SegWit marker and flag
            has_witness = False
            if offset + 1 < len(tx_bytes) and tx_bytes[offset] == 0x00 and tx_bytes[offset + 1] == 0x01:
                has_witness = True
                offset += 2  # Skip marker (0x00) and flag (0x01)

            # Decode inputs
            tx_inputs: list[TransactionInput] = []
            # Input count (varint)
            tx_input_count, offset = self.decode_varint(tx_bytes, offset)
            for _ in range(tx_input_count):
                tx_inputs, offset = self.decode_transaction_inputs_from_raw_tx(tx_bytes.hex())

            # Decode outputs
            tx_outputs: list[TransactionOutput] = []
            # Output count (varint)
            tx_output_count, offset = self.decode_varint(tx_bytes, offset)
            for _ in range(tx_output_count):
                tx_outputs, offset = self.decode_transaction_outputs_from_raw_tx(tx_bytes.hex())

            # Decode witnesses
            tx_witnesses: list[TransactionWitness] = []
            if has_witness:
                # Witness count (varint)
                tx_witness_count, offset = self.decode_varint(tx_bytes, offset)
                for _ in range(tx_witness_count):
                    tx_witnesses, offset = self.decode_transaction_witnesses_from_raw_tx(tx_bytes.hex())

            # Lock time (4 bytes)
            lock_time = struct.unpack('<I', tx_bytes[offset:offset + 4])[0]
            offset += 4

            # Calculate total output value
            total_output_value = sum(tx_output['value_satoshi'] for tx_output in tx_outputs)

            decoded_tx: Transaction = {
                'version': version,
                'tx_type': 'SegWit' if has_witness else 'Legacy',
                'has_witness': has_witness,
                'tx_input_count': tx_input_count,
                'tx_inputs': tx_inputs,
                'tx_output_count': tx_output_count,
                'tx_outputs': tx_outputs,
                'lock_time': lock_time,
                'size_bytes': offset,
                'total_output_value_satoshi': total_output_value,
                'total_output_value_btc': total_output_value / 100000000.0,
                'tx_witnesses': tx_witnesses
            }

        except ValueError as e:
            raise struct.error(f"Failed to unpack transaction data: {e}") from e

        return decoded_tx

    # --------------------------------------------------------------------------------------------
    def extract_raw_txs(self, body: str, offset: int = 0) -> list[str]:
        """
        """

        # Validate data
        match body:
            case str() if body.strip():
                body = body.strip().lower()
            case _:
                raise ValueError("Block body data must be a non-empty string")

        # Validate hex string format and convert to bytes
        try:
            body_bytes: bytes = bytes.fromhex(body)
        except ValueError as ex:
            raise ValueError(f"Invalid hex string: {ex}") from ex

        txs: list[str] = []

        try:
            # read transaction count
            tx_count, offset = self.decode_varint(body_bytes, offset)

            for _ in range(tx_count):
                start_offset = offset

                # --- parse just enough to find tx end ---
                version = body_bytes[offset:offset + 4]
                offset += 4

                has_witness = False
                if body_bytes[offset] == 0x00 and body_bytes[offset + 1] == 0x01:
                    has_witness = True
                    offset += 2

                # inputs
                tx_inputs_count, offset = self.decode_varint(body_bytes, offset)

                for _ in range(tx_inputs_count):
                    offset += 32  # prev txid
                    offset += 4  # index
                    script_len, offset = self.decode_varint(body_bytes, offset)
                    offset += script_len
                    offset += 4  # sequence

                # outputs
                tx_outputs_count, offset = self.decode_varint(body_bytes, offset)
                for _ in range(tx_outputs_count):
                    offset += 8  # value
                    script_len, offset = self.decode_varint(body_bytes, offset)
                    offset += script_len

                # witnesses
                if has_witness:
                    for _ in range(tx_inputs_count):
                        n_stack, offset = self.decode_varint(body_bytes, offset)
                        for _ in range(n_stack):
                            item_len, offset = self.decode_varint(body_bytes, offset)
                            offset += item_len

                # locktime
                offset += 4

                # slice raw tx
                tx_end = offset
                txs.append(body_bytes[start_offset:tx_end].hex())
        except ValueError as ex:
            raise ValueError(f"Error extracting raw txs: {ex}") from ex

        return txs

    # --------------------------------------------------------------------------------------------
    def decode_transaction_inputs_from_raw_tx(self, tx: str, offset: int = 0) -> tuple[list[TransactionInput], int]:
        """
        """

        # Validate and convert input
        if not isinstance(tx, str) or not tx.strip():
            raise ValueError("Transaction must be a non-empty string")

        try:
            tx_bytes = bytes.fromhex(tx.strip())
        except ValueError as e:
            raise ValueError(f"Invalid hex string: {e}") from e

        if len(tx_bytes) < 10:  # Minimum transaction size
            raise ValueError("Transaction too short to be valid")

        try:
            # Skip version (4 bytes)
            offset += 4

            # Check for SegWit marker and flag
            has_witness = False
            if offset + 1 < len(tx_bytes) and tx_bytes[offset] == 0x00 and tx_bytes[offset + 1] == 0x01:
                has_witness = True
                offset += 2  # Skip marker (0x00) and flag (0x01)

            # Input count
            tx_inputs_count, offset = self.decode_varint(tx_bytes, offset)

            tx_inputs: list[TransactionInput] = []

            for _ in range(tx_inputs_count):
                prev_tx_hash = tx_bytes[offset:offset + 32][::-1].hex()
                offset += 32

                prev_output_index = struct.unpack("<I", tx_bytes[offset:offset + 4])[0]
                offset += 4

                script_len, offset = self.decode_varint(tx_bytes, offset)

                script_sig = tx_bytes[offset:offset + script_len].hex()
                offset += script_len

                sequence = struct.unpack("<I", tx_bytes[offset:offset + 4])[0]
                offset += 4

                is_coinbase = (prev_tx_hash == "00" * 32 and prev_output_index == 0xFFFFFFFF)

                tx_inputs.append({
                    "previous_tx_hash": prev_tx_hash,
                    "previous_output_index": prev_output_index,
                    "script_sig": script_sig,
                    "script_sig_length": hex(script_len),
                    "sequence": hex(sequence),
                    "is_coinbase": is_coinbase,
                })
        except (struct.error, IndexError) as e:
            raise ValueError(f"Failed to decode transaction: {e}") from e
        return tx_inputs, offset

    # -------------------------------------------------------------------------------------------------
    def decode_transaction_outputs_from_raw_tx(self, tx: str, offset: int = 0) -> tuple[list[TransactionOutput], int]:
        """
        """

        # Validate and convert input
        if not isinstance(tx, str) or not tx.strip():
            raise ValueError("Transaction must be a non-empty string")

        try:
            tx_bytes = bytes.fromhex(tx.strip())
        except ValueError as e:
            raise ValueError(f"Invalid hex string: {e}") from e

        if len(tx_bytes) < 10:  # Minimum transaction size
            raise ValueError("Transaction too short to be valid")

        try:
            # Skip version (4 bytes)
            offset += 4

            # Check for SegWit marker and flag
            has_witness = False
            if offset + 1 < len(tx_bytes) and tx_bytes[offset] == 0x00 and tx_bytes[offset + 1] == 0x01:
                has_witness = True
                offset += 2  # Skip marker (0x00) and flag (0x01)

            # Input count and skip inputs
            tx_inputs_count, offset = self.decode_varint(tx_bytes, offset)

            # Skip all inputs
            for _ in range(tx_inputs_count):
                # Skip previous tx hash (32 bytes) + output index (4 bytes)
                offset += 36

                # Skip script sig
                script_pubkey_length, offset = self.decode_varint(tx_bytes, offset)
                offset += script_pubkey_length

                # Skip sequence (4 bytes)
                offset += 4

            # Output count
            tx_outputs_count, offset = self.decode_varint(tx_bytes, offset)

            tx_outputs: list[TransactionOutput] = []

            for i in range(tx_outputs_count):
                if offset + 8 > len(tx_bytes):
                    raise ValueError(f"Insufficient data for output {i} value")

                # Value (8 bytes, little-endian)
                value_satoshi = struct.unpack("<Q", tx_bytes[offset:offset + 8])[0]
                offset += 8

                # Script length and script
                script_pubkey_length, offset = self.decode_varint(tx_bytes, offset)

                if offset + script_pubkey_length > len(tx_bytes):
                    raise ValueError(f"Insufficient data for output {i} script")

                script_pubkey = tx_bytes[offset:offset + script_pubkey_length].hex()
                offset += script_pubkey_length

                block_tx_output: TransactionOutput = {
                    "value_satoshi": value_satoshi,
                    "value_btc": value_satoshi / 100000000.0,
                    "script_pubkey": script_pubkey,
                    "script_pubkey_length": script_pubkey_length
                }

                tx_outputs.append(block_tx_output)

        except (struct.error, IndexError) as e:
            raise ValueError(f"Failed to decode transaction: {e}") from e

        return tx_outputs, offset

    def decode_transaction_witnesses_from_raw_tx(self, tx: str) -> tuple[list[TransactionWitness], int]:
        """
        """
        # Validate and convert input
        if not isinstance(tx, str) or not tx.strip():
            raise ValueError("Transaction must be a non-empty string")

        try:
            tx_bytes = bytes.fromhex(tx.strip())
        except ValueError as e:
            raise ValueError(f"Invalid hex string: {e}") from e

        if len(tx_bytes) < 10:  # Minimum transaction size
            raise ValueError("Transaction too short to be valid")

        offset = 0

        try:
            # Skip version (4 bytes)
            offset += 4

            # Check for SegWit marker + flag
            if not (offset + 1 < len(tx_bytes) and
                    tx_bytes[offset] == 0x00 and
                    tx_bytes[offset + 1] == 0x01):
                raise ValueError("Transaction is not SegWit format (no witness marker/flag)")

            offset += 2  # Skip marker and flag

            # Input count
            input_count, offset = self.decode_varint(tx_bytes, offset)

            # Skip all inputs
            for _ in range(input_count):
                # Skip previous tx hash (32 bytes) + output index (4 bytes)
                offset += 36

                # Skip script sig
                script_len, offset = self.decode_varint(tx_bytes, offset)
                offset += script_len

                # Skip sequence (4 bytes)
                offset += 4

            # Skip outputs
            output_count, offset = self.decode_varint(tx_bytes, offset)

            for _ in range(output_count):
                # Skip value (8 bytes)
                offset += 8

                # Skip script pubkey
                script_len, offset = self.decode_varint(tx_bytes, offset)
                offset += script_len

            # Now decode witness data
            witnesses: list[TransactionWitness] = []

            for input_index in range(input_count):
                # Number of witness items for this input
                witness_count, offset = self.decode_varint(tx_bytes, offset)

                witness_items: list[WitnessItem] = []
                total_witness_size = 0

                # Decode each witness item
                for item_index in range(witness_count):
                    # Length of this witness item
                    item_len, offset = self.decode_varint(tx_bytes, offset)

                    if offset + item_len > len(tx_bytes):
                        raise ValueError(f"Insufficient data for witness item {item_index} of input {input_index}")

                    # Extract witness item data
                    item_data = tx_bytes[offset:offset + item_len]
                    offset += item_len

                    # Analyze the witness item
                    witness_item = self.decode_witness_item(item_data, item_index, witness_count)
                    witness_items.append(witness_item)
                    total_witness_size += item_len

                # Determine witness type
                witness_type = self.determine_witness_type(witness_count, witness_items)

                witnesses.append(TransactionWitness(
                    input_index=input_index,
                    witness_count=witness_count,
                    witness_items=witness_items,
                    witness_type=witness_type,
                    total_witness_size=total_witness_size,
                ))

        except (struct.error, IndexError) as e:
            raise ValueError(f"Failed to decode transaction witnesses: {e}") from e

        return witnesses, offset

    # -------------------------------------------------------------------------------------------------
    def decode_witness_item(self, data: bytes, index: int, witness_count: int) -> WitnessItem:
        """
        """
        hex_data = data.hex()
        length = len(data)

        # Empty witness item
        if length == 0:
            return WitnessItem(
                data="",
                length=0,
                type="EMPTY",
                description="Empty witness item"
            )

        # For P2WPKH (2 items: signature + pubkey)
        if witness_count == 2:
            if index == 0:
                # First item should be signature (typically 70-72 bytes with DER encoding + sighash)
                if 70 <= length <= 73:
                    sighash_type = data[-1] if length > 0 else 0
                    sighash_desc = self.get_sighash_description(sighash_type)
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="SIGNATURE",
                        description=f"ECDSA signature ({length} bytes) - {sighash_desc}"
                    )
                else:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="SIGNATURE",
                        description=f"ECDSA signature ({length} bytes) - non-standard length"
                    )
            elif index == 1:
                # Second item should be public key (33 bytes compressed or 65 bytes uncompressed)
                if length == 33:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="PUBLIC_KEY",
                        description="Compressed public key (33 bytes)"
                    )
                elif length == 65:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="PUBLIC_KEY",
                        description="Uncompressed public key (65 bytes)"
                    )
                else:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="PUBLIC_KEY",
                        description=f"Public key ({length} bytes) - non-standard length"
                    )

        # For P2WSH (multisig or script)
        elif witness_count > 2:
            if index == 0:
                # First item is typically empty for multisig due to OP_CHECKMULTISIG bug
                if length == 0:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="EMPTY",
                        description="Empty item (OP_CHECKMULTISIG dummy)"
                    )
            elif index == witness_count - 1:
                # Last item is typically the redeem script
                return WitnessItem(
                    data=hex_data,
                    length=length,
                    type="REDEEM_SCRIPT",
                    description=f"Witness script/redeem script ({length} bytes)"
                )
            else:
                # Middle items are typically signatures
                if 70 <= length <= 73:
                    sighash_type = data[-1] if length > 0 else 0
                    sighash_desc = self.get_sighash_description(sighash_type)
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="SIGNATURE",
                        description=f"ECDSA signature ({length} bytes) - {sighash_desc}"
                    )
                else:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="SIGNATURE",
                        description=f"ECDSA signature ({length} bytes) - non-standard length"
                    )

        # Single witness item (unusual)
        elif witness_count == 1:
            return WitnessItem(
                data=hex_data,
                length=length,
                type="UNKNOWN",
                description=f"Single witness item ({length} bytes) - possibly custom script"
            )

        # Default case
        return WitnessItem(
            data=hex_data,
            length=length,
            type="UNKNOWN",
            description=f"Unknown witness data ({length} bytes)"
        )

    # -------------------------------------------------------------------------------------------------
    @staticmethod
    def determine_witness_type(witness_count: int, witness_items: list[WitnessItem], /) -> str:
        """
        Determine the overall witness type based on witness items.

        Args:
            witness_count: Number of witness items
            witness_items: List of analyzed witness items

        Returns:
            String describing the witness type
        """
        match witness_count:
            case 0:
                return "NO_WITNESS"
            case 1:
                return "CUSTOM_SCRIPT"
            case 2:
                # P2WPKH pattern: signature + pubkey
                if (len(witness_items) == 2 and
                        witness_items[0].get("type") == "SIGNATURE" and
                        witness_items[1].get("type") == "PUBLIC_KEY"):
                    return "P2WPKH"
                else:
                    return "UNKNOWN_2_ITEM"
            case count if count > 2:
                # P2WSH multisig pattern: empty + signatures + script
                if (witness_items and
                        witness_items[0].get("type") == "EMPTY" and
                        witness_items[-1].get("type") == "REDEEM_SCRIPT"):
                    sig_count = sum(1 for item in witness_items[1:-1]
                                    if item.get("type") == "SIGNATURE")
                    return f"P2WSH_MULTISIG_{sig_count}_OF_N"
                elif witness_items and witness_items[-1].get("type") == "REDEEM_SCRIPT":
                    return "P2WSH_CUSTOM_SCRIPT"
                else:
                    return f"COMPLEX_WITNESS_{count}_ITEMS"
            case _:
                return "UNKNOWN_WITNESS"

    # -------------------------------------------------------------------------------------------------
    @staticmethod
    def get_sighash_description(sighash_type: int, /) -> str:
        """
        Get description of sighash type.

        Args:
            sighash_type: Sighash type byte

        Returns:
            Human-readable description of sighash type
        """
        base_type = sighash_type & 0x1f
        anyonecanpay = bool(sighash_type & 0x80)

        match base_type:
            case 0x01:
                base_desc = "SIGHASH_ALL"
            case 0x02:
                base_desc = "SIGHASH_NONE"
            case 0x03:
                base_desc = "SIGHASH_SINGLE"
            case _:
                base_desc = f"SIGHASH_UNKNOWN({base_type:02x})"

        if anyonecanpay:
            return f"{base_desc} | SIGHASH_ANYONECANPAY"
        else:
            return base_desc

    # --------------------------------------------------------------------------------------------
    @staticmethod
    def decode_varint(data: bytes, offset: int) -> tuple[int, int]:
        """
        Decode a variable-length integer from the data starting at offset.

        Args:
            data (bytes): The byte data
            offset (int): Starting position

        Returns:
            tuple: (value, new_offset)
        """
        first_byte = data[offset]
        if first_byte < 0xFD:
            return first_byte, offset + 1
        elif first_byte == 0xFD:
            return struct.unpack("<H", data[offset + 1: offset + 3])[0], offset + 3
        elif first_byte == 0xFE:
            return struct.unpack("<I", data[offset + 1: offset + 5])[0], offset + 5
        elif first_byte == 0xFF:
            return struct.unpack("<Q", data[offset + 1: offset + 9])[0], offset + 9
        else:
            raise ValueError("Invalid varint prefix byte")


# -------------------------------------------------------------------------------------------------
def main():
    block_decoder: BlockDecoder = BlockDecoder()

    # set block height here
    block_height = 0  # change this to any block height you want

    url = f"https://blockchain.info/rawblock/{block_height}?format=hex"
    resp = requests.get(url)

    if resp.status_code == 200:
        block = resp.text.strip()  # raw block hex
        block_bytes = bytes.fromhex(block)
        decoded_block = block_decoder.decode_full_block(block_bytes.hex())
        print(json.dumps(decoded_block, indent=2))
    else:
        print("Error:", resp.status_code, resp.text)


if __name__ == "__main__":
    main()

Piensa por ti mismo para decidir.

1. Lo que quieres.
2. Lo que es verdad.
3. Lo que hay que hacer para obtener (1) a la luz de (2)

Hazlo con humildad y apertura de miras para tener en cuenta las mejores ideas de las que dispones.

Busca siempre patrones de las cosas que te afectan, para entender las relaciones causa-efecto que las impulsan y extraer principios que te sirvan para afrontarlas de forma eficaz.

1. Admite y afronta la realidad.

1.1 Sé hiperrealista.

a. Sueños + realidad + determinación = una vida de éxito.

1.2 La verdad y una comprensión acertada de la realidad es la base primordial de cualquier resultado positivo.

1.3 Ten la mente radicalmente abierta. Actúa con total transparencia.

a. La transparencia y la apertura de miras radicales son imprescindibles para aprender rápido y cambiar de verdad.

b. No dejes que el miedo al qué dirán se interponga en tu camino.

c. La sinceridad y la transparencia radical atraerán tareas y relaciones más significativas.

1.4 Observa la naturaleza para comprender el funcionamiento de la realidad.

a. No te obsesiones con tu visión de cómo deberían ser las cosas, o no aprenderás como son en realidad.

b. Para ser bueno, un hecho debe operar en consonancia con las leyes de la realidad y contribuir al desarrollo.

c. La evolución es la fuerza más poderosa del universo, la única permanente y la que lo impulsa todo.

d. Renuévate o muere.

1.5 La evolución es el mayor logro y la mayor recompensa que puede haber en la vida.

a. Los incentivos del individuo han de estar alineados con los objetivos del grupo.

b. La realidad opera en beneficio del conjunto, no del individuo.

c. La adaptación mediante el ensayo y error rápidos es de un valor incalculable.

d. Eres todo y nada a la vez. Decide que quieres ser.

e. Lo que eres depende de tu punto de vista.

1.6 Entiende las lecciones de la naturaleza.

a. Maximiza tu progreso.

b. Recuerda que sin sacrificio no hay progreso.

c. Para fortalecerse, la ley natural dicta que uno ha de exceder sus limites, y eso duele.

1.7 Dolor + reflexión = progreso

a. No rehúyas el dolor, ábrele los brazos.

b. Acepta el amor exigente.

1.8 Sopesa las consecuencias de segundo y tercer orden.

1.9 Atente a las consecuencias de tus actos.

1.10 Observa la maquinaria desde arriba.

a. Piensa en tí mismo como una máquina dentro de otra, y sé consciente de la capacidad para alterar ambas y obtener mejores resultados.

b. Comparando los resultados y los objetivos, puedes decidir cómo modificar la maquinaria.

c. Distingue entre tú como diseñador de tu maquinaria y como operario de esta.

d. El craso error de la mayoría de las personas es que no se ven a sí mismas ni a los demás de forma objetiva, y eso les lleva a tropezar repetidamente con las debilidades propias y ajenas.

e. Son triunfadores quienes pueden sobreponerse a sí mismos, para mirar las cosas con objetividad y aprovecharse de ellas a fin de propiciar un cambio.

f. Pedir ayuda a aquellos cuyas fuerzas se complementan con tus debilidades es una habilidad que ha de desarrollarse a toda costa, por cuanto implicará la construcción de barreras que te impedirán hacer lo que no debas.

g. Como es difícil observarse a uno mismo objetivamente, has de confiar en la opinión de los demás y en cualquier hecho demostrable.

h. Con una mente lo bastante abierta y decidida, puedes lograr prácticamente cualquier objetivo.

2. Utiliza el proceso de 5 pasos para obtener lo que quieras de la vida.

2.1 Ten objetivos claros.

a. Prioriza: Cuando se puede lograr prácticamente todo, es imposible conseguir nada.

b. No confundas metas con deseos.

c. Decidir lo que quieres en la vida consiste en conciliar objetivos y deseos.

d. No confundas las señales de éxito con el propio éxito.

e. No descartes un objetivo por considerarlo inalcanzable.

f. Recuerda que con un liston alto se desarrollan grandes capacidades.

g. Nada te detendrá si eres a) flexible b) autónomo.

h. Aceptar cómo lidiar con tus tropiezos es tan importante como seguir adelante.

2.2 Identifica y no tolores los problemas.

a. Considera los problemas dolorosos como mejoras en potencia que te hacen señas evidentes.

b. No evites afrontar los problemas porque provienen de realidades desagradables que no es grato afrontar.

c. Identifica los problemas de manera especifica.

d. No confundas la causa de un problema con el problema en si mismo.

e. Distingue los problemas grandes de los pequeños.

f. Una vez identificado el problema, no lo toleres.

2.3 Diagnostica los problemas para llegar a sus causas últimas.

a. Primero debes saber qué pasa y luego decidir qué vas a hacer.

b. Distingue las causas primeras de las causas últimas.

c. Admite que si conoces a alguien (tambien a ti mismo) sabrás qué esperar de él.

2.4 Traza un plan.

a. Retrocede antes de avanzar.

b. Considera los problemas como una serie de resultados producidos por una maquina.

c. Recuerda que normalmente hay muchos caminos para llegar a un mismo fin.

d. Piensa en un plan como en un guión cinematográfico que muestre qué debe hacer cada uno a los largo del tiempo.

e. Considera poner tu plan por escrito, para que todos lo vean y usen como baremo de tu progreso.

f. Reconoce que no necesitas demasiado tiempo para diseñar un buen plan.

2.5 Transforma las estrategias en resultados.

a. Los buenos planificadores que no ponen en práctica sus ideas van sin rumbo.

b. Los buenos habitos de trabajo se subestiman demasiado.

c. Establece parámetros claros para asegurarte de que sigues el plan.

2.6 Recuerda que las flaquezas no importan si les encuentras soluciones.

a. Busca patrones en los errores e identifica qué paso.

b. Cada cual tiene un obstáculo principal que se interpone entre él y él éxito; debes encontrar el tuyo y afrontarlo debidamente.

2.7 Comprende tu esquema mental y tu humildad, así como los ajenos.

3. Sé radicalmente abierto de miras.

3.1 Reconoce las dos barreras.

a. Entiende la barrera del ego.

b. Los dos yos y su lucha por el control.

c. Entiende la barrera del punto ciego.

3.2 Practica una apertura de miras radical.

a. Cree sinceramente que, quizá, deconozcas cuál es el mejor camino, y reconoce que la capacidad de aceptar esa ignorancia es mas importante que todo lo que sepas.

b. Reconoce que las decisiones se toman en dos etapas: primero se recopila toda la información relevante y luego se decide.

c. No te preocupes por la apariencias, sino por el objetivo.

d. Reconoce que es imposible producir sin aprender.

e. Reconoce que, para ponerte efectivamente en la piel del otro y ver el mundo desde su perspectiva, has de de proceder a una suspención temporal del juicio: solo desde la empatía podrás evaluar correctamente una perspectiva ajena.

f. Recuerda que buscas la mejor respuesta, no la mejor respuesta que puedas llegar por tu cuenta.

g. Deja claro si estás discutiendo o si quieres aprender y endenter. Piensa en qué opción es mejor, según tu credibilidad propia y la de los demás.

3.3 Aprecia el arte del desacuerdo reflexivo.

3.4 Triangula tu perspectiva con gente creíble dispuesta a discrepar.

a. Hacer planes para que el peor caso factible sea lo mejor posible.

3.5 Reconoce los sintomas de la cerrazón y la apertura de miras y préstales atención.

3.6 Entiende cómo se puede llegar a ser radicalmente abierto de miras.

a. Utiliza el dolor con regularidad como guía hacia unas reflexiones de calidad.

b. Haz de la apertura de miras una constumbre.

c. Conoce tus puntos ciegos.

d. Si varias personas creíbles te dicen que estas obrando mal, y eres el único que no lo ve así, da por supuesto que seguramente no estés siendo neutral.

e. Medita.

f. Básate en los hechos y anima a los demás a que hagan los mismo.

g. Haz todo lo que esté en tus manos para ayudar a los demás a ser tambien abiertos de miras.

h. Utilza una herramienta de decisión basada en los hechos.

i. Comprende cuándo es mejor rendirse y tener fe en tu sistema de toma de decisiones.

4. Entiende que las personas funcionan de formas muy distintas.

4.1 Entiende el poder que se deriva de comprender cómo funcionas tú y los demás.

a. Naces con atributos que pueden ser una ayuda o un escollo, según cómo los utilices.

4.2 El trabajo y las relaciones significativas no son solo cosas bonitas que escoges, sino que las llevas programadas genéticamente.

4.3 Entiende las grandes batallas mentales y cómo controlarlas para obtener lo que quieres.

a. Entiende que la mente consciente está enzarzada en una batalla con la mente inconsciente.

b. Sé consciente de que la luca más constante que existe es la de los sentimientos y el pensamiento.

c. Compagina los sentimientos y el pensamiento.

d. Escoge bien tus hábitos.

e. Instruye al yo inferior de forma afable y persistente para desarrollar buenos hábitos.

f. Entiende la diferencia entre el hemisferio derecho y el izquierdo, y el pensamiento derivado de cada uno.

g. Entiende lo mucho y lo poco que el cerebro puede cambiar.

[AWS] Tagging Resources

Amazon Web Services allows customers to assign metadata to their AWS resources in the form of tags. Each tag is a simple label consisting of a customer-defined key and an optional value that can make it easier to manage, search for, and filter resources by purpose, owner, environment, or other criteria. AWS tags can be used for many purposes.

Important

Do not add personally identifiable information (PII) or other confidential or sensitive information in tags. Tags are accessible to many AWS services, including billing. Tags are not intended to be used for private or sensitive data.

Best practices

As you create a tagging strategy for AWS resources, follow best practices:

• Do not store personally identifiable information (PII) or other confidential or sensitive information in tags.

• Use a standardized, case-sensitive format for tags, and apply it consistently across all resource types.

• Consider tag guidelines that support multiple purposes, like managing resource access control, cost tracking, automation, and organization.

• Use automated tools to help manage resource tags. AWS Resource Groups and the Resource Groups Tagging API enable programmatic control of tags, making it easier to automatically manage, search, and filter tags and resources.

• Use too many tags rather than too few tags.

• Remember that it is easy to change tags to accommodate changing business requirements, but consider the consequences of future changes. For example, changing access control tags means you must also update the policies that reference those tags and control access to your resources.

Best Practices for Naming Tags

Adopt a Standardized Approach for Tag Names

Keep in mind that names for AWS tags are case sensitive so ensure that they are used consistently. For example, the tags CostCenter and costcenter are different, so one might be configured as a cost allocation tag for financial analysis and reporting and the other one might not be. Similarly, the Name tag appears in the AWS Console for many resources, but the name tag does not.

A number of tags are predefined by AWS or created automatically by various AWS services. Many AWS-defined tags are named using all lowercase, with hyphens separating words in the name, and prefixes to identify the source service for the tag.

For example:

• aws:ec2spot:fleet-request-id identifies the Amazon EC2 Spot Instance Request that launched the instance

• aws:cloudformation:stack-name identifies the AWS CloudFormation stack that created the resource

• lambda-console:blueprint identifies blueprint used as a template for an AWS Lambda function

• elasticbeanstalk:environment-name identifies the application that created the resource

Important

Consider naming your tags using all lowercase, with hyphens separating words, and a prefix identifying the organization name or abbreviated name.

For example:

• anycompany:cost-center to identify the internal Cost Center code

• anycompany:environment-type to identifywhether the environment is development, test, or production

• anycompany:application-id to identify the application the resource was created for

The prefix ensures that tags are clearly identified as having been defined by your organization and not by AWS or athird-party tool that you may be using. Using all lowercase with hyphens for separators avoids confusion about how to capitalize a tag name. For example, anycompany:project-idis simpler to rememberthan ANYCOMPANY:ProjectID, anycompany:projectID, or Anycompany:ProjectId

Standardize Names for AWS Resources

[AWS] Naming Conventions Best Practices

Tagging categories

Technical Tags:

1. name: Used to give a descriptive name to the resource. For example, “web-server-prod” or “db-dev”.
2. owner: Specifies the person or team responsible for the resource. This can be an individual’s name, team name, or email address.
3. availability-zone: Specifies the availability zone or region where the resource is located.
4. service: Indicates the AWS service associated with the resource, such as “ec2”, “s3”, “rds”, etc.
5. lifecycle: Specifies the lifecycle stage of the resource, such as “active”, “inactive”, “archived”.
6. version: Indicates the version of the resource if applicable.
7. created-date: Specifies the date when the resource was created.
8. last-modified-date: Specifies the date when the resource was last modified.
9. resource-id: Unique identifier for the resource.
10. environment-type: Specifies the type of environment, such as “test”, “qa”, or “production”.

Tags for Automation:

1. automated: Indicates whether the resource is managed by automation tools. Values can be “yes” or “no”.
2. auto-scaling-group: Indicates the auto-scaling group associated with the resource.
3. automation-tool: Specifies the automation tool used to manage the resource, such as “cloudformation”, “terraform”, etc.
4. scheduled-automation: Indicates whether the resource is part of scheduled automation tasks. Values can be “yes” or “no”.
5. continuous-integration: Indicates whether the resource is integrated with a continuous integration (CI) system.
6. automated-backup: Specifies whether automated backups are enabled for the resource.
7. auto-shutdown: Indicates whether the resource has an automated shutdown schedule.
8. auto-recovery: Specifies whether the resource is configured for automatic recovery in case of failure.
9. auto-scaling-policy: Indicates the auto-scaling policy associated with the resource.
10. automated-monitoring: Specifies whether automated monitoring is enabled for the resource.

Business Tags:

1. cost-center: Associates the resource with a specific cost center or project code for financial tracking and allocation.
2. department: Indicates the department or business unit that owns or uses the resource. Useful for chargebacks and cost allocation.
3. project: Associates the resource with a specific project or initiative within the organization.
4. business-unit: Specifies the business unit that the resource belongs to.
5. customer-id: Identifies the customer or client associated with the resource.
6. revenue-center: Associates the resource with a specific revenue center or profit center.
7. business-criticality: Indicates the business criticality of the resource, such as “high”, “medium”, or “low”.
8. contract-id: Specifies the contract ID related to the resource.
9. service-level-agreement: Indicates the service level agreement (SLA) associated with the resource.
10. business-impact: Specifies the potential business impact of the resource being unavailable.

Security Tags:

1. security-classification: Specifies the security classification or sensitivity level of the resource, such as “public”, “internal”, “confidential”.
2. compliance: Specifies the compliance status of the resource, such as “hipaa”, “pci-dss”, “gdpr”, etc.
3. backup: Indicates whether the resource is included in regular backups or not. Values can be “yes” or “no”.
4. encryption: Indicates whether the resource data is encrypted at rest and/or in transit.
5. access-control-list: Specifies the access control list (ACL) associated with the resource.
6. security-group: Indicates the security group associated with the resource.
7. firewall-rule: Specifies the firewall rule associated with the resource.
8. vulnerability: Indicates the vulnerability status of the resource, such as “vulnerable”, “patched”, etc.
9. data-classification: Specifies the data classification level of the resource, such as “sensitive”, “confidential”, etc.
10. security-policy: Indicates the security policy applied to the resource.

Tags for Classification:

1. environment: Indicates the environment the resource belongs to, such as “production”, “development”, or “staging”.
2. application: Identifies the application or project to which the resource belongs. Useful for multi-application environments.
3. region: Specifies the AWS region where the resource is located.
4. zone: Indicates the availability zone within a region where the resource is located.
5. role: Specifies the role or function of the resource within the system architecture.
6. service-type: Specifies the type of service provided by the resource, such as “web”, “database”, etc.
7. deployment-stage: Indicates the deployment stage of the resource, such as “alpha”, “beta”, “release-candidate”, etc.
8. cost-allocation: Specifies the cost allocation tag for financial tracking purposes.
9. cost-savings: Indicates whether the resource is part of a cost-saving initiative. Values can be “yes” or “no”.
10. expiration-date: Specifies the date when the resource should be reviewed or decommissioned. Useful for managing resource lifecycle.

Tagging Use Cases

Tags for AWS Console Organization and Resource Groups

• organize-resources: Use tags to group and categorize resources based on their purpose, project, environment, or any other relevant attribute. This makes it easier to navigate and manage your resources within the AWS Management Console.
• resource-group-classification: Leverage tags to create resource groups based on specific criteria, such as tagging all production resources together or grouping resources belonging to a specific department or business unit.

Tags for Cost Allocation

• cost-tracking: Assign tags to AWS resources to identify the associated costs and allocate them to specific cost centers, departments, projects, or customers. This enables you to track and analyze your AWS spending accurately.
• cost-allocation-reports: Utilize tags to filter and generate detailed cost allocation reports using AWS Cost Explorer or Cost and Usage Reports. These reports can provide insights into cost breakdowns based on different tag values.

Tags for Automation

• automated-resource-management: Use tags to trigger automation scripts, workflows, or AWS Lambda functions for managing resources automatically. For example, you can use tags to automatically start or stop EC2 instances based on predefined schedules or resource utilization.
• automated-backup-and-recovery: Tagging resources can be useful in automated backup and recovery scenarios. By assigning specific tags to resources, you can create backup policies or automation scripts that target resources with those tags for backup or recovery operations.

Tags for Operations Support

• resource-monitoring-and-alerts: Assign tags to resources to facilitate monitoring and alerting systems. Tags can be used to set up monitoring dashboards, define alarm rules, or enable automated actions based on resource state changes or performance metrics.
• incident-response-and-troubleshooting: Tags can assist in incident response and troubleshooting by providing additional context or categorization for resources. During an incident, tags can help identify affected resources, their interdependencies, or associated configurations.

Tags for Access Control

• access-control-policies: Use tags to define fine-grained access control policies through AWS Identity and Access Management (IAM). Tags can be used as condition keys in IAM policies, allowing you to grant or restrict access to resources based on their tag values.
• resource-sharing-and-cross-account-access: Leverage tags to control resource sharing and enable cross-account access. You can use tags to specify which resources are accessible to other AWS accounts or specific IAM roles based on tag-based permissions.

Tags for Security Risk Management

• security-group-and-nacl-rules: Assign tags to security groups and network ACLs (NACLs) to manage and audit security policies effectively. Tags can help categorize or identify the purpose of security rules, allowing you to track and analyze security configurations and potential risks.
• asset-inventory-and-vulnerability-management: Utilize tags to maintain an inventory of your AWS assets and support vulnerability management practices. Tags can help identify resources that require regular security assessments, vulnerability scanning, or patch management.

Creating a VPC without using the wizard

In this post we are going to create a VPC with a private and public subnet without using the wizard to get into VPC in detail and to know how it works internally.

You also can create your VPC using the wizard following the AWS tutorial Creating a VPC with Public and Private Subnets for Your Compute Environments

1. Open the AWS console and go to Services → Networking & Content Delivery section and make click on VPC.

2. On the left menu, in the VIRTUAL PRIVATE CLOUD section, click on Your VPCs option.

3. Click on the button that says Create VPC on the top side.

4. Fill the form that appears with the data below.

   Name tag: vpc-us-east-1-test-lab
   IPv4 CIDR block: 10.0.0.0/16
   IPv6 CIDR block: Amazon provided IPv6 CIDR block
   Tenancy: Default

   Note Be aware of the VPC description that appears in the form. A VPC is an isolated portion of the AWS cloud populated by AWS objects, such as Amazon EC2 instances. You must specify an IPv4 address range for your VPC. Specify the IPv4 address range as a Classless Inter-Domain Routing (CIDR) block; for example, 10.0.0.0/16. You cannot specify an IPv4 CIDR block larger than /16. You can optionally associate an IPv6 CIDR block with the VPC.

   Note By default, when you create a custom VPC, the following components are created.

   • A router
   • A main route table
   • A main ACL
   • A main Security Group

5. On the VPC section, verify that the VPC was created successfully and then make click on Subnets on the left menu.

6. Click on the button that says Create subnet on the top side.

7. Fill the form that appears with the data below.

   Name tag: subnet-us-east-1a-public-test-lab
   VPC: vpc-us-east-1-test-lab
   Availability Zone: us-east-1a
   IPv4 CIDR block: 10.0.1.0/24
   IPv6 CIDR block: Don’t Assign IPv6

8. Repeat the steps 5 and 6 in order to create a second subnet.

9. Fill the form that appears with the data below.

   Name tag: subnet-us-east-1b-private-test-lab
   VPC: vpc-us-east-1-test-lab
   Availability Zone: us-east-1b
   IPv4 CIDR block: 10.0.2.0/24
   IPv6 CIDR block: Don’t Assign

10. Go to the Subnet section and select subnet-us-east-1b-private-test-lab, then click on Actions button on the top side, then click on Modify auto-assign IP settings and finally select Auto-assign IPv4 option.

11. Go to the VPC Dashboard section and on the left menu click on Internet Gateways and then click on Create internet gateway on the top side.

12. Fill the form that appears with the data below.

    Name tag: igw-vpc-us-east-1-test-lab

13. Attach the internet gateway to the VPC vpc-us-east-1-test-lab

14. Go to the VPC Dashboard section and on the left menu click on Route Tables and then click on Create route table on the top side.

15. Fill the form that appears with the data below.

    Name tag: rtb-subnet-us-east-1a-public-test-lab
    VPC: vpc-us-east-1-test-lab

16. Repeat the steps 14 and 15 in order to create a route table for the private subnet.

    Name tag: rtb-subnet-us-east-1b-private-test-lab
    VPC: vpc-us-east-1-test-lab

17. Select the route table rtb-subnet-us-east-1a-public-test-lab and on the bottom side select routes and then click on the Edit routes button.

18. Add the following rules and then click save.

    Destination	Target
    0.0.0.0/0	igw-vpc-us-east-1-test-lab
    ::/0	igw-vpc-us-east-1-test-lab

19. Select the route table rtb-subnet-us-east-1a-public-test-lab and on the bottom side select subnet associations and then click on Edit subnet associations, select subnet-us-east-1a-public-test-lab and save.

20. Create a NAT gateway associated with the subnet subnet-us-east-1a-public-test-lab and add a tag name as ng-subnet-us-east-1a-public-test-lab

21. Select the route table rtb-subnet-us-east-1b-private-test-lab and on the bottom side select routes and then click on the Edit routes button.

22. Add the following rules and then click save.

    Destination	Target
    0.0.0.0/0	ng-subnet-us-east-1a-public-test-lab

23. Select the route table rtb-subnet-us-east-1b-private-test-lab and on the bottom side select subnet associations and then click on Edit subnet associations, select subnet-us-east-1b-private-test-lab and save.

24. Go to the VPC Dashboard section and on the left menu click on Network ACLs and then click on Create network ACL on the top side.

25. Create two ACL one for the public subnet and another for the private subnet.

    [ACL-1]
    Name tag: acl-subnet-us-east-1a-public-test-lab
    VPC: vpc-us-east-1-test-lab

    [ACL-2]
    Name tag: acl-subnet-us-east-1b-private-test-lab
    VPC: vpc-us-east-1-test-lab

26. Associate each ACL with its corresponding subnet.

27. Add to each ACL the rules you need.

28. Create two Security Groups one for the public subnet and another for the private subnet.

    [SG-1]
    Security group name: sg-subnet-us-east-1a-public-test-lab
    Description: Any description
    VPC: vpc-us-east-1-test-lab

    [SG-2]
    Security group name: sg-subnet-us-east-1b-private-test-lab
    Description: Any description
    VPC: vpc-us-east-1-test-lab

References and Resources

• Amazon Virtual Private Cloud Documentation
• Amazon Virtual Private Cloud VPC
• CIDR

AWS Naming Conventions Best Practices

A naming convention is a well-defined set of rules useful for choosing the name of an AWS resource. Ensure that your AWS resources are using appropriate naming conventions for tagging in order to manage them more efficiently and adhere to AWS resource tagging best practices.

Naming (tagging) your AWS resources consistently have several advantages such as providing additional information about the resource location and usage, promoting consistency within the selected AWS region, distinguishing fast similar resources stacks from one another, avoiding naming collisions, improving clarity in cases of potential ambiguity and enhancing aesthetic and professional appearance.

Default Pattern Components

{RegionCode}
(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)

{AvailabilityZoneCode}
([1-2]{1})([a-c]{1})

{EnvironmentCode}
(dev|test|stg|prod)

{ApplicationCode}
([a-z0-9\-]+)

{SubnetRouteCode}
(public|private)

{ResourceCode}
(ec2|alb|nlb)

VPC Naming Conventions

Default Pattern Format

vpc-{RegionCode}-{EnvironmentCode}-{ApplicationCode}

RegExp

^vpc-(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)-(dev|test|stg|prod)-([a-z0-9\-]+)$

Examples

vpc-us-east-1-prod-bigdataappstack
vpc-us-west-2-prod-webappstack

Subnet Naming Conventions

Default Pattern Format
subnet-{RegionCode}-{AvailabilityZoneCode}-{SubnetRouteCode}-{EnvironmentCode}-{ApplicationStackCode}

RegExp
^subnet-(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)-([1-2]{1})([a-c]{1})-(public|private)-(dev|test|stg|prod)-([a-z0-9\-]+)$

Examples
subnet-us-east-1-2a-public-prod-webappstack
subnet-us-west-1-2b-private-prod-databasestack

Security Group Naming Conventions

Default Pattern Format
sgr_{ResourceCode}_{RegionCode}_{EnvironmentCode}-{ApplicationCode}

RegExp
^sgr_(ec2|alb|nlb)_(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)_(dev|test|stg|prod)_([a-z0-9\-]+)$

Examples
sgr_ec2_us-west-1_dev_sample-app-instance-1

EC2 Instance Naming Conventions

Default Pattern Format
ec2_{RegionCode}_{AvailabilityZoneCode}_{EnvironmentCode}_{ApplicationCode}

RegExp
^ec2_(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)_([1-2]{1})([a-c]{1})_(dev|test|stg|prod)_([a-z0-9\-]+)$

Examples
ec2_us-east-1_2a_prod_tomcat
ec2_us-west-1_2b_prod_nodejs

PEM Key Naming Conventions

Default Pattern Format
key-RegionCode-EnvironmentCode-ApplicationCode

RegExp
^key-(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)-(dev|test|stg|prod)-([a-z0-9\-]+)$

Examples
key-us-east-1-prod-web-server
key-us-west-1-prod-database

ECS Cluster Naming Conventions

Default Pattern Format
ecs-cluster_{RegionCode}_{EnvironmentCode}_{ApplicationCode}

RegExp
^ecs-cluster_(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)_(dev|test|stg|prod)_([a-z0-9\-]+)$

Examples
ecs-cluster_us-east-1_prod_tomcat
ecs-cluster_us-east-1_prod_micro-app

ECS Task Definition Naming Conventions

Default Pattern Format
ecstd-{RegionCode}-{EnvironmentCode}-{ApplicationCode}-{Vesion}

RegExp
^ecstd-(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)-(dev|test|stg|prod)-([a-z0-9\-]+)-v1$

Examples
ecstd-us-east-1-prod-tomcat-v1
ecstd-us-east-1-prod-micro_app-v1

ECS Service Naming Conventions

Default Pattern Format
ecs-service_{RegionCode}_{EnvironmentCode}_{ApplicationCode}

RegExp
^ecs-service_(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)_(dev|test|stg|prod)_([a-z0-9\-]+)$

Examples
ecs-service_us-east-1_prod_tomcat
ecs-service_us-east-1_prod_micro-app

References

• Cloud Conformity VPCs
• Cloud Conformity EC2

Apache Kafka Architecture & Fundamentals Q&A

These are the answers and questions from a tech talk led by Confluent about Apache Kafka Architecture and Fundamentals.

How reliable is the data ingest to Kafka from the producer(s)?
Data can be acknowledged in the broker and also in copies (replicas) on other brokers. You have your choice of no ack, ack from the receiving broker, or ack once that one and all of the broker copies have successfully received the message.

Does Kafka require Hadoop cluster infrastructure like MapReduce, HDFS, etc.?
No, a Kafka cluster of brokers requires only Kafka, and for now, Zookeeper

Can we store in local storage or also on data lake storage?
Kafka can store on both - in memory initially and persisted on disk at an adjustable time after.

Is the retention time the same for all brokers in a cluster? Or Can we set a different retention time for each broker?
Retention time can be set per topic, not per broker.

Is there any way to keep the data rather than a generic retention time configuration setting? For example, if there is a slower moving set of topics compared to higher velocity data
Yes - retention times are uniquely configurable based on the particular Kafka architectural design.

If data expires within Broker’s local storage then does it mean it get lost? Suppose it has not been read/consumed by “all” the consumers of that data
Kafka doesn’t really support the concept of a specific piece of data having been consumed. It may be consumed by multiple consumers - or if your retention period is short and no consumers are consuming from the topic it is in, yes a particular record could be deleted without ever being consumed.

Kafka brokers needs a separated master?

I’m not sure what you mean by this question. Brokers work cooperatively in a cluster. At this time, they use Zookeeper for some coordination, although that requirement will be removed in the future.

Are there local disks SSD or hard drives?
Local disks for the broker can be of any type.

Can some other components be used instead of Zookeeper?
No but Zookeeper is set to be deprecated at some point - please see KIP 500: https://www.confluent.io/blog/removing-zookeeper-dependency-in-kafka/

Can Kafka be used in on-premise setup? If so what is the minimum system requirements/how many servers are required to configure Kafka?
Kafka can be run on as few as one server, but three would commonly be ideal for failover support.

Is it kind of a layer in between source and target to stream the data directly without doing any data transformation?
I’m a little unclear on the intent of this question. Yes, data can pass through Kafka without any transformation. Because the producers of the data and the consumers of the data are decoupled, and data is persisted on the cluster, data can be consumed by multiple different consumers over time. So data could be transported to many targets.

If a message is successfully consumed, it is stored till the retention time any way?
Yes! Once received, messages are immutable until the retention period expires.

Does the data stay on local storage for 5 days ?
The retention time, the time that data is preserved on local storage, is configurable by topic - so data could stay on one topic for 5 days and another for a much longer or shorter time. It is not uncommon to have transient data - that is just being passed through Kafka, to be retained for only a few hours. Some businesses store data on some topics for much longer - weeks, months, years or forever - depending on their business needs.

Is there a limit to the number of topics that a Broker node can handle?
There are best practices - but this is more of a logical limit, i.e., how many topics to keep track of. There needs to be enough, but not too many depending on the applications using Kafka.

Is Zookeeper platformed within a Kafka instance or do I have to stand up and operate a separate zookeeper instance to keep my cluster operational?
Yes, Zookeeper needs to be stood up when installing Kafka currently - but as Joe said it will be going away per KIP 500. Please see: https://www.confluent.io/blog/removing-zookeeper-dependency-in-kafka/

Can we have authorization enabled for consumer connecting using rest proxy having SASL_SSL security with SCRAM mechanism.?
This mechanism is for authentication - for authorization, you need to use ACLs (Access Control Lists) or the new RBAC (Role Based Access Control) which is a feature of the Confluent Platform, not open source Kafka.

Can kafka be used as permanent data store?
Yes! Kafka can be used as a permanent store, ksqlDB allows it to even be used as a form of a database!

Can we use a 3rd party enterprise metadata instead of the zookeeper?
At this time, Zookeeper is required. In the near future, Kafka clusters will be self managing and the Zookeeper requirement will be removed - but will not need any other 3rd party tools.

What does adding consumers mean? Does adding consumer mean adding a new node to the cluster?
Adding consumers means adding more applications that are reading from the cluster (brokers).

When kafka store data on local disk , what is preferred 1) one disk with 15tb of size or 2) 15 disks with 1tb size each?
OF course, that depends. Kafka is a distributed event system, so you can have multiple Kafka Brokers and then each will need to have its own local disk or disks.

Is it common to have a consumer that is also a producer? For example, a process that takes incoming messages and “enriches” them for later consumption by a different process?
Yes, it is common - in fact, any stream processing application both consumes data from Kafka and then producers transformed data back to Kafka.

If we request to produce or consume.. will it go to zookeeper or broker?
All clients - producers and consumers - communicate only with Kafka Brokers.

Can a message be consumed more than once by the same consumer?
Generally an individual consumer will only consume any message one time - however, a consumer can set its consumer offset - the point at which it is reading from a topic - and could then re-read any number of messages.

How is KAFKA different from a Message Queue MQ Series or ESB solutions?
The main difference is independence between the producer of messages and consumers of messages. The broker is in the middle as an independent handler of either 1) producer write or 2) consumer read requests in a given transaction.

What are the data formats that are supported by Kafka? Also where does the transformations happen for business rules?
Kafka supports many data formats - there are many serializers of data types to include string, long, int, etc. Also there are many Schema formats to include Avro, Protobuf, and JSON Schema.

Any chance of message will be lost in Kafka?
It is possible for a message to be lost when produced to Kafka - you should look at the producer ACK configuration settings. If a producer sets ACKS=0, then it will just send a message and not care if it is received by a Broker or not.

I understand that the storage is always held by Zookeeper. Is it true or Can I set for instance Oracle as the persistence?
Message data in a Kafka cluster is only persisted on local Kafka Broker storage. It is not held by Zookeeper nor can you persist it to Oracle. You could, of course use Connect to move data from Kafka storage to Oracle if you wish.

Any scenarios where we don’t need Zookeeper or we should always use it? We have an environment implemented by consultants that does not use Zookeeper. There are also some articles on the Internet that talks about not using Zookeeper.
With the current release of Kafka, you must use Zookeeper. That requirement will be removed in a future release as Kafka clusters move to self management.

If a broker dies, is the producer data on broker lost, or is it replicated on other brokers as well so it is never lost?
When you create a topic, you define the amount of replication that topic requires. It is typical to have 3 replicas for a topic - which means that even if 2 brokers crash you can continue to produce to and consume from your topic without data loss.

Is retention time the only way to have data removed/dropped from the cluster or can it be set in a way that if a consumer has consumed the data then it can be dropped from the cluster?
Kafka doesn’t keep track of whether data has been consumed or not - there can be multiple consumers doing different tasks with the same data, or one consumer may choose to re-consume an entire topic - so retention time controls how long data will be stored on Kafka - on a per topic basis.

Can I add in Kafka code to enrich new data with some topic? Or need do outside in another process?
Yes! This is perfect for a Streams application if more than just a minimal addition is being done.

Would it be possible to stream the messages of size 25MB via kafka? Any limitations?
The default maximum message size on Kafka is 1 MB. While you can configure Kafka to use larger messages, it is not optimal to do so.

Can two producers write to the same Topic? Or is Topic specific to a Producer?
Yes - that is part of the beauty of Kafka! Multiple producers. And the same with consumers too! :-)

Is the kafka architecture based generally enough to handle multiple use cases, or is the architecture very bespoke depending on each customer use case?
Excellent question! For basic applications, it is general enough - but for more complicated ones, that for example may need to scale, it would need to become bespoke. Rather, the larger the Kafka architecture the more bespoke it generally needs to be.

What data producers will not connect to kafka?
To produce to Kafka, your producer needs to conform to Kafka protocols. The native APIs are Java based, but there are libraries for many other languages. If you are not able to use any of the available languages, you can send data to Kafka through the REST Proxy, which can communicate with any process that can send an HTTP REST request. You can also use Kafka Connect to move data from many standard data sources into Kafka.

Does data also split the data in between multiple partitions?
A Kafka Producer will be able to send data to any partition of a topic - from the metadata it receives from a Kafka Broker, it knows where the leader broker is for each partition of a topic and it uses a partitioner to determine which partition to send data to.

How to decide the partition size for a topic ? What happens if we resize the topic once it is created with “N” partitions?
Great question that has a lot behind it. There are some White Papers and/or blog posts on the Confluent and Kafka websites that help address this. Much of what Joe is explaining is the basis for that! Yes, it’s possible to resize as an operation, but with some caveats.

If I need to post transactions extracted from 500 tables into kafka, 1.is that a good idea to create 500 topics for each table, to make sure the consumers reads it in the same insertion order
Good point! Order is something that requires overhead to enforce in Kafka. So yes, you may end up needing to create individual topics to ensure order. But for what you reference, please check out Kafka Connect which may be able help handle that better.

Presenter mentioned that partitions are striped across brokers, but I thought broker cluster used leader and standby replicas? Are the brokers not replicated, but working in concert?
Each topic may be broken into a number of partitions - you define the number when you create a topic. Those partitions can reside on different brokers - and each broker will receive data appropriate for its own partitions. Partitions can also be replicated - so each partition will have a leader broker and some number of follower brokers.

Are there any advantages of using just one single topic, collecting all those 500 table messages? I am afraid this would mix up the insertion order when consumers read 500 tables data from one single topic.
Yes, to ensure order for sure you can use single topics each having single partitions. But there may be a way to design your application, depending on its needs, that could be more efficient - please check out the white papers and blog entries on confluent.io!

Compaction retention policy does or does not allow for mutation of messages?
Messages are not mutated once they arrive at a Kafka broker - that is an inherent part of the design - all messages are immutable. However, messages can be removed from a topic if it is compacted - but the content of the message will not change.

Can I change the sequences of the messages in the logs?
No - and yes. No you can’t change what’s been written to a log as it’s immutable. But you can further process what’s been written into a log, where you read it - and maybe even process it or add to it from say a DB - then write it to a different log. Pretty cool?! This is known as a Streams application.

So, the data written by producers need to be schema less right ? And consumer needs to understand that format right?
The data needs to be serialized to Kafka on the way in (produce), and deserialized on the way out (consume). You can use schemas to help control this.

How many offsets we can have in a stream?
The offset number for a message in a partition is a 64 bit number, so you can have 2 to the 64th power number of messages in a partition. You can define your topic to have many partitions. A stream can have as many messages as the underlying topic.

Can I run Kafka on raspberry pi with small memory foot print. What is it’s memory foot print. I assuming memory foot print is configurable .please confirm.
Yes! Kafka will run on Raspberry Pi. Sorry I don’t have the memory requirements, you should be able to find them at https://kafka.apache.org/ and please also check out http://confluent.io !

Is there a chance of messages being out of sync, means the timestamp is not in order?
As a producer you can define what the timestamp is - you could use the event time, when the event actually took place, or the time the message is produced by the producer or the ingest time, when the message is written by the Broker. So, yes, you could have messages in a topic that have timestamps that are out of order.

Is a log file equivalent to a segment? Partition has 1 to n log files
Raphael, yes! log file = segment.

How to avoid multiple deliveries of same messages?
Simply you can turn on idempotence in your Kafka client code. You can also use Exactly Once Semantics - also known as EOS: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

Can we use Kafka to process a large cuantiti of daily images, pdf documents as messages and storage?
Because the default message size is 1 MD in Kafka, it is not unusual to have image or pdf data directly in the messages. That said, you could chunk large messages - but then you are responsible for correctly reassembling them. Or, you can pass references in a message to images or pdf documents that are stored externally.

How do you typically handle duplicate messages? Let’s say we need to do some operation say write to a DB, but a consumer can receive the same message twice..And we aren’t able to know whether the item is already written to the DB or not when we receive the duplicate msg.
Please check this out - explains a lot of what I think you may want to know: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

Are there any limitations if we use Python ? Does streaming support Python?
Python is a supported language for writing Kafka producer and consumer code. However Java is the closest to Kafka’s source code.

How to create an AWS billing alarm

1. Set the billing preference, check and enable the option to receive billing alerts.
2. Go to Cloud Watch console.
3. Change the region, make sure your AWS region is settled to US EAST (N. Virginia).
4. Create an alarm, click on Alarms on the left side of the panel and in the dashboard that pops up click on the orange button that says “Create alarm”.
5. Select a metric, in the panel that pops up, click the button that says “Select metric”. This will pop up a window that allows you to choose which type of service you would like to use. Select Billing
6. Select the service, in the next window you can choose the total estimated charged on your account.
7. Select the period of time that the alarm will be activated.
8. Set the threshold.
9. Create a notification using SNS, you just click on Create new topic give it a name and enter the email where you want to receive the notification.
10. Add a description.
11. Click on Preview and create

Simple Storage Service (S3)

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics.

Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements.

Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.

You can use S3 to store and retrieve any amount of data at any time, from anywhere on the web.

The basics of S3 are

• S3 is object-based and allow you to upload files.
• Files can be from 0 Bytes to 5TB
• There is unlimited storage
• Files are store in buckets. Think about bucket as folders.
• S3 is a universal namespace, this means that the name of the bucket must be unique globally.
• When you upload a file to S3, you will receive a http 200 code if the upload was successful

S3 Objects

Think of objects just as files. Objects consist of the following:

• Key: This is simply the name of the object.
• Value: This is simply the data and is made up of a sequence of Bytes.
• VersionId: Used for versioning
• Metadata: Data about the data you are storing
• Sub-resources: ACLs, torrent

S3 data consistency

• Read after write consistency for PUTS of new objects, this means that if you upload a file for first time you will be able to read it immediately.

• Eventual consistency for overwrite PUTS and DELETES, this means that if you upload or update or delete an existing file and try to read it immediately you may get the older version or you may not, basically changes to objects can take a little bit of time to propagate.

S3 features

• Tired storage availability (S3 standard, S3-IA, S3-OneZone-IA, S3 Intelligent Tiering, S3 Glacier, S3 Glacier deep archive).
• Lifecycle management.
• Versioning.
• Encryption.
• MFA for delete.
• Secure your data using access control lists ACLs and bucket policies.

S3 storage classes and tiers.

• S3 Standard: 99.99% availability and 99.99999999999% durability, store your objects redundantly across multiple devices in multiple facilities and is designed to sustain the loss of two facilities concurrently.

• S3-IA Infrequent Access: Is for data that is less frequently accessed but requires rapid access when needed. It has a lower fee than S3 Standard, but you will be charged a retrieval fee.

• S3 One Zone Infrequent Access: For where you want a low-cost option for infrequently accessed data, but that does not require the multiple availability zone data resilience.

• S3 Intelligent-Tiering: Designed to optimize costs by automatically moving data to the most cost-effective access tier, without performance impact or operational overhead.

• S3 Glacier: Is a secure, durable and low-cost storage class for data archiving. You can reliably store any amount of data at costs that are competitive with or cheaper than on-premise solutions. Retrieval times is configurable from minutes to hours.

• S3 Glacier Deep Archiving: Is the amazon S3 lower-cost storage class where a retrieval time of 12 hours is acceptable.

S3 performance

Your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned prefix. There are no limits to the number of prefixes in a bucket. You can increase your read or write performance by using parallelization. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second. Similarly, you can scale write operations by writing to multiple prefixes.

Optimizing Amazon S3 performance

S3 Security and Encryption

By default, all newly created bucket are private, you can setup access control to your bucket using:

• Bucket policies at bucket level.
• Access Control List ACLs at objects level.

S3 buckets can be configured to create access logs which log all requests made to the S3 bucket. This can be sent to another bucket and even another bucket in another account.

Encryption in transit

Encryption in transit is achieved by SSL/TLS

Encryption at rest (Server Side)

• S3 Manage Keys SS3-S3: Where Amazon manages all the keys
• AWS Key Management Service SSE-KMS: Where you and Amazon manage the keys
• Server Side Encryption Where you provide the KEY-SSE-C and give to Amazon your own keys that you manage.

Encryption at rest (Client Side)

Where you upload your files already encrypted using your own keys.

S3 Versioning

• Stores all versions of an object (Including all writes and even if you delete an object)
• Once enabled, versioning cannot be disabled but only suspended.
• Versioning is integrated with lifecycle rules.
• Versioning has MFA capabilities, which uses multi-factor authentication to provide an additional layer of security.

S3 Lifecycle management

Uses lifecycle rules to manage your objects, which define how Amazon S3 manages objects during lifetime. Lifecycle rules enable you to automatically transition objects among the different storage classes and tier from S3 to Glacier. Using a lifecycle rule you can automatically expire objects based on your retention needs or clean up incomplete multi-parts uploads.

Sharing S3 buckets and objects across accounts.

There three ways to share buckes across accounts.

• Using bucket policies and IAM Applies across the entire bucket and provide programmatic access only.
• Using ACLs and IAM Applies to individual objects and provide programmatic access only.
• Cross-Account IAM Roles Provides programmatic and console access.
• See Example walkthroughs: Managing access to your amazon S3 resources

S3 Cross region replication

Replication enables automatic, asynchronous copying of objects across Amazon S3 buckets. Buckets that are configured for object replication can be owned by the same AWS account or by different accounts. You can copy objects between different AWS Regions or within the same Region.

• Versioning must be enabled on both the source and the destination buckets.
• Files in an existing bucket are not replicated automatically.
• All subsequent updated files will be replicated automatically.
• Delete markers are not replicated.
• Deleting individual versions or delete markers will not be replicated.

To enable object replication, you add a replication configuration to your source bucket. The minimum configuration must provide the following:

• The destination bucket where you want Amazon S3 to replicate objects
• An AWS Identity and Access Management (IAM) role that Amazon S3 can assume to replicate objects on your behalf

S3 Transfer acceleration.

S3 transfer acceleration uses the Cloud Front edge network to accelerate your uploads to S3 instead of uploading directly to your S3 bucket, you can use a distinct URL to upload your files to an edge location which then transfer that files to S3 using the Amazon enhanced network.

Enables fast, easy and secure transfer of files over long distances between your clients and an S3 bucket. S3 transfer acceleration takes advantage of Amazon CloudFront globally distributed edge locations as the data arrives at an edge location and then is routed to an S3 bucket over an optimized network path.

You will get a distinct URL to upload your files, example: buckt-name.s3-accelerate.amazonaws.com

How is S3 billed

• For storage.
• For requests.
• For storage management.
• For data transfer.
• For transfer acceleration.
• For cross-region replication.

AWS Identity and Access Management (IAM)

Is a web service that helps you securely control access to AWS resources. You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources. IAM allows you to manage users and their level of access to the aws console.

IAM Key terminology

• Users End user such as people, employees of an organization, etc.
• Groups A collection of users. Each user in the group will inherit the permissions of the group.
• Policies Policies are made of documents, called policy documents. These documents are in JSON format and they give permissions as to what a user/group is able to do.
• Roles You create roles a then assign them to AWS resources.

IAM Tips

• IAM is universal. It does not apply to regions at this time.
• The root account is simply the account created when first setup your AWS account. It has complete admin access.
• New users has NO permissions when first created.
• New users are assigned and Access Key and an Secret Key when first created.
• You cannot use the access key and the secret key to login into the console.
• You can view the access key and the secret key once, if you lose it you have to regenerate them.
• Always set up Multifactor Authentication especially for the root account.
• You can create and customise your own password rotation policies.

IAM Best practices

• Lock Away Your AWS Account Root User Access Keys You use an access key (an access key ID and secret access key) to make programmatic requests to AWS. However, do not use your AWS account root user access key. The access key for your AWS account root user gives full access to all your resources for all AWS services, including your billing information. You cannot reduce the permissions associated with your AWS account root user access key.

• Create Individual IAM Users Don’t use your AWS account root user credentials to access AWS, and don’t give your credentials to anyone else. Instead, create individual users for anyone who needs access to your AWS account. Create an IAM user for yourself as well, give that user administrative permissions, and use that IAM user for all your work. For information about how to do this, see Creating Your First IAM Admin User and Group.

• Use Groups to Assign Permissions to IAM Users Instead of defining permissions for individual IAM users, it’s usually more convenient to create groups that relate to job functions (administrators, developers, accounting, etc.). Next, define the relevant permissions for each group. Finally, assign IAM users to those groups.

• Grant Least Privilege When you create IAM policies, follow the standard security advice of granting least privilege, or granting only the permissions required to perform a task. Determine what users (and roles) need to do and then craft policies that allow them to perform only those tasks.

• Get Started Using Permissions with AWS Managed Policies To get started quickly, you can use AWS managed policies to give your employees the permissions they need to get started. These policies are already available in your account and are maintained and updated by AWS. For more information about AWS managed policies, see AWS Managed Policies.

• Use Customer Managed Policies Instead of Inline Policies For custom policies, we recommend that you use managed policies instead of inline policies. A key advantage of using these policies is that you can view all of your managed policies in one place in the console. You can also view this information with a single AWS CLI or AWS API operation. Inline policies are policies that exist only on an IAM identity (user, group, or role). Managed policies are separate IAM resources that you can attach to multiple identities. For more information, see Managed Policies and Inline Policies.

• Use Access Levels to Review IAM Permissions When you review a policy, you can view the policy summary that includes a summary of the access level for each service within that policy. AWS categorizes each service action into one of five access levels based on what each action does: List, Read, Write, Permissions management, or Tagging. You can use these access levels to determine which actions to include in your policies.

• Configure a Strong Password Policy for Your Users If you allow users to change their own passwords, require that they create strong passwords and that they rotate their passwords periodically. On the Account Settings page of the IAM console, you can create a password policy for your account. You can use the password policy to define password requirements, such as minimum length, whether it requires non-alphabetic characters, how frequently it must be rotated, and so on. For more information, see Setting an Account Password Policy for IAM Users.

• Enable MFA For extra security, we recommend that you require multi-factor authentication (MFA) for all users in your account.

• Use Roles for Applications That Run on Amazon EC2 Instances Applications that run on an Amazon EC2 instance need credentials in order to access other AWS services. To provide credentials to the application in a secure way, use IAM roles. A role is an entity that has its own set of permissions, but that isn’t a user or group.

• Use Roles to Delegate Permissions Don’t share security credentials between accounts to allow users from another AWS account to access resources in your AWS account. Instead, use IAM roles. You can define a role that specifies what permissions the IAM users in the other account are allowed. You can also designate which AWS accounts have the IAM users that are allowed to assume the role.

• Do Not Share Access Keys Access keys provide programmatic access to AWS. Do not embed access keys within unencrypted code or share these security credentials between users in your AWS account. For applications that need access to AWS, configure the program to retrieve temporary security credentials using an IAM role. To allow your users individual programmatic access, create an IAM user with personal access keys.

• Rotate Credentials Regularly Change your own passwords and access keys regularly, and make sure that all IAM users in your account do as well.

• Remove Unnecessary Credentials Remove IAM user credentials (passwords and access keys) that are not needed.

• Use Policy Conditions for Extra Security To the extent that it’s practical, define the conditions under which your IAM policies allow access to a resource. For example, you can write conditions to specify a range of allowable IP addresses that a request must come from. You can also specify that a request is allowed only within a specified date range or time range. You can also set conditions that require the use of SSL or MFA (multi-factor authentication). For example, you can require that a user has authenticated with an MFA device in order to be allowed to terminate an Amazon EC2 instance. For more information, see IAM JSON Policy Elements: Condition in the IAM Policy Elements Reference.

• Monitor Activity in Your AWS Account You can use logging features in AWS to determine the actions users have taken in your account and the resources that were used. The log files show the time and date of actions, the source IP for an action, which actions failed due to inadequate permissions, and more.

• Video Presentation About IAM Best Practices

References

• AWS Identity And Access Management

You may have heard that Bitcoin is based on cryptography, which is a branch of mathematics used extensively in computer security. Cryptography means “secret writing” in Greek, but the science of cryptography encompasses more than just secret writing, which is referred to as encryption. Cryptography can also be used to prove knowledge of a secret without revealing that secret (digital signature), or prove the authenticity of data (digital fingerprint). These types of cryptographic proofs are the mathematical tools critical to bitcoin and used extensively in bitcoin applications.

Ironically, encryption is not an important part of bitcoin, as its communications and transaction data are not encrypted and do not need to be encrypted to protect the funds. mastering bitcoin

ECDSA (‘Elliptic Curve Digital Signature Algorithm’) is the cryptography behind private and public keys used in Bitcoin. It consists of combining the math behind finite fields and elliptic curves to create one way equations, meaning you can choose your private key “Pick a number between 1 and \(2^{256}\).” and easily calculate your public key (some other number). However, You can not take the public key and easily calculate their private key. In fact, for Bitcoin it would take trillions of computers trillions of years of continuous guessing of different private keys to figure out which one creates a given public key. How does ECDSA work in Bitcoin

Elliptic curve cryptography (ECC)

Elliptic curve cryptography (ECC) is based on the discrete logarithm problem that is based on elliptic curves over finite fields (Galois fields). The main benefit of ECC over other types of public key algorithms is that it needs a smaller key size while providing the same level of security as, f or example, RSA.

An elliptic curve will simply be the set of points described by the equation:

(this is required to exclude singular curves) The equation above is what is called Weierstrass normal form for elliptic curves.

If we want to explicitly take into account the point at infinity, we can refine our definition of elliptic curve as follows:

Algebraic addition

Algebraic formulae are constructed to efficiently compute the geometric arithmetic.

Adding distinct points P and Q

Note that m is the slope of the line through P and Q.

Doubling the point P

When P and Q share the same position (X and Y wise), addind two points (P and Q ) that share the same location is called “Doubling the point P” also referred as adding a point to itself.

Recall that a is one of the parameters chosen with the elliptic curve and that m is the tangent on the point P.

Elliptic curve point multiplication

Elliptic curve point multiplication is the operation of successively adding a point along an elliptic curve to itself repeatedly. It is used in elliptic curve cryptography (ECC) as a means of producing a one-way function. The literature presents this operation as scalar multiplication, thus the most common name is elliptic curve scalar multiplication, as written in Hessian form of an elliptic curve. wikipedia

Point addition and point doubling allow us to define scalar multiplication for elliptic curves such that, xP = R where x is a scalar, P is a point tangent to the curve and R is the resulting point from adding P on to itself x times.

The straightforward way of computing a point multiplication is through repeated addition. However, this is a fully exponential approach to computing the multiplication.

There are several algorithms used to calculate the scalar multiplication:

• Elliptic curve point multiplication

• Fast point multiplication algorithms for binary elliptic curveswith and without precomputation

Elliptic Curve Digital Signature Algorithm (ECDSA)

Bitcoin uses the Elliptic Curve Digital Signature Algorithm (ECDSA) based on elliptic curve cryptography. The particular elliptic curve is known as secp256k1, which is the curve Elliptic curve cryptography (ECC).

The elliptic curve domain parameters over Fp associated with a Koblitz curve secp256k1 are specified by the sextuple

T = (p,a,b,G,n,h) where the finite field Fp is defined by:

p = FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFE FFFFFC2F
= 2^256 - 2^32 - 2^9 - 2^8 - 2^7 - 2^6 - 24 - 1

a = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
b = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000007

The base point G in compressed form is:

G = 02 79BE667E F9DCBBAC 55A06295 CE870B07 029BFCDB 2DCE28D9 59F2815B 16F81798

and in uncompressed form is:

G = 04 79BE667E F9DCBBAC 55A06295 CE870B07 029BFCDB 2DCE28D9 59F2815B 16F81798 483ADA77 26A3C465 5DA4FBFC 0E1108A8 FD17B448 A6855419 9C47D08F FB10D4B8

Finally the order n of G and the cofactor are:

n = FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFE BAAEDCE6 AF48A03B BFD25E8C D0364141
h = 01

secp256k1 has characteristic p, it is defined over the prime field ℤp. Some other curves in common use have characteristic 2, and are defined over a binary Galois field GF(2n), but secp256k1 is not one of them.

As the a constant is zero, the ax term in the curve equation is always zero, hence the curve equation becomes \(\displaystyle y^{2} \ =\ x^{3}+7\)

Private and Public Keys

A bitcoin wallet contains a collection of key pairs, each consisting of a private key and a public key. The private key (k) is a number, usually picked at random. mastering bitcoin

Private Keys

A private key is simply a number, picked at random. Ownership and control over the private key is the root of user control over all funds associated with the corresponding bitcoin address. The private key is used to create signatures that are required to spend bitcoin by proving ownership of funds used in a transaction. The private key must remain secret at all times, because revealing it to third parties is equivalent to giving them control over the bitcoin secured by that key. The private key must also be backed up and protected from accidental loss, because if it’s lost it cannot be recovered and the funds secured by it are forever lost, too. mastering bitcoin

Generating a private key from a random number

The first and most important step in generating keys is to find a secure source of entropy, or randomness. Creating a bitcoin key is essentially the same as “Pick a number between 1 and \(2^{256}\).” The exact method you use to pick that number does not matter as long as it is not predictable or repeatable. Bitcoin software uses the underlying operating system’s random number generators to produce 256 bits of entropy (randomness). Usually, the OS random number generator is initialized by a human source of randomness, which is why you may be asked to wiggle your mouse around for a few seconds. mastering bitcoin

More precisely, the private key can be any number between 0 and n - 1 inclusive, where n is a constant (n = 1.1578 * 1077, slightly less than \(2^{256}\)) defined as the order of the elliptic curve used in bitcoin. mastering bitcoin

Private Key Wallet import format

1 - Take a private key

0C28FCA386C7A227600B2FE50B7CAE_SAMPLE_PRIVATE_KEY_DO_NOT_IMPORT_11EC86D3BF1FBE471BE89827E19D72AA1D

2 - Add a 0x80 byte in front of it for mainnet addresses or 0xef for testnet addresses. Also add a 0x01 byte at the end if the private key will correspond to a compressed public key

800C28FCA386C7A227600B2FE50B7C_SAMPLE_PRIVATE_KEY_DO_NOT_IMPORT_AE11EC86D3BF1FBE471BE89827E19D72AA1D

3 - Perform SHA-256 hash on the extended key

8147786C4D15106333BF278D71DADAF1079EF2D2440A4DDE37D747DED5403592

4 - Perform SHA-256 hash on result of SHA-256 hash

507A5B8DFED0FC6FE8801743720CEDEC06AA5C6FCA72B07C49964492FB98A714

5 - Take the first 4 bytes of the second SHA-256 hash, this is the checksum

507A5B8D

6 - Add the 4 checksum bytes from point 5 at the end of the extended key from point 2

800C28FCA386C7A227600B2FE50B7CAE11EC8_SAMPLE_PRIVATE_KEY_DO_NOT_IMPORT_6D3BF1FBE471BE89827E19D72AA1D507A5B8D

7 - Convert the result from a byte string into a base58 string using Base58Check encoding. This is the Wallet Import Format

5HueCGU8rMjxEXxiPuD5BDk_SAMPLE_PRIVATE_KEY_DO_NOT_IMPORT_u4MkFqeZyd4dZ1jvhTVqvbTLvyTJ

Sample Java Code

    try {
      // 1 - Take a private key
      String privateKey = "0C28FCA386C7A227600B2FE50B7CAE11EC86D3BF1FBE471BE89827E19D72AA1D";

      // 2 - Add a 0x80 byte in front of it for mainnet addresses or 0xef for testnet addresses.
      // Also add a 0x01 byte at the end if the private key will correspond to a compressed public key
      String privateKeyWIF = "80" + privateKey;

      // 3 - Perform SHA-256 hash on the extended key
      MessageDigest sha256 = MessageDigest.getInstance("SHA-256");
      byte[] sha1 = sha256.digest(DatatypeConverter.parseHexBinary(privateKeyWIF));
      //log.info("Sha1: {}", bytesToHex(sha1).toUpperCase());

      // 4 - Perform SHA-256 hash on result of SHA-256 hash
      byte[] sha2 = sha256.digest(sha1);
      //log.info("Sha2: {}", bytesToHex(sha2).toUpperCase());

      // 5 - Take the first 4 bytes of the second SHA-256 hash, this is the checksum
      String checksum = bytesToHex(sha2).substring(0, 8).toUpperCase();
      //log.info("Checksum: {}", checksum);

      // 6 - Add the 4 checksum bytes from point 5 at the end of the extended key from point 2
      privateKeyWIF = privateKeyWIF + checksum;
      //log.info("Private Key + Checksum: {}", privateKeyWIF.toUpperCase());

      // 7 - Convert the result from a byte string into a base58 string using Base58Check encoding. This is the Wallet Import Format
      privateKeyWIF = Base58.encode(hexToBytes(privateKeyWIF));
      //log.info("Private Key WIF: {}", privateKeyWIF);
    } catch (NoSuchAlgorithmException e) {
      e.printStackTrace();
      return null;
    }

Generating Public Key and Address

0 - Having a private ECDSA key

18e14a7b6a307f426a94f8114701e7c8e774e7f9a47e2c2035db29a206321725

1 - Take the corresponding public key generated with it (33 bytes, 1 byte 0x02 (y-coord is even), and 32 bytes corresponding to X coordinate)

0250863ad64a87ae8a2fe83c1af1a8403cb53f53e486d8511dad8a04887e5b2352

2 - Perform SHA-256 hashing on the public key

0b7c28c9b7290c98d7438e70b3d3f7c848fbd7d1dc194ff83f4f7cc9b1378e98

3 - Perform RIPEMD-160 hashing on the result of SHA-256

f54a5851e9372b87810a8e60cdd2e7cfd80b6e31

4 - Add version byte in front of RIPEMD-160 hash (0x00 for Main Network)

00f54a5851e9372b87810a8e60cdd2e7cfd80b6e31

5 - Perform SHA-256 hash on the extended RIPEMD-160 result

ad3c854da227c7e99c4abfad4ea41d71311160df2e415e713318c70d67c6b41c

6 - Perform SHA-256 hash on the result of the previous SHA-256 hash

c7f18fe8fcbed6396741e58ad259b5cb16b7fd7f041904147ba1dcffabf747fd

7 - Take the first 4 bytes of the second SHA-256 hash. This is the address checksum

c7f18fe8

8 - Add the 4 checksum bytes from stage 7 at the end of extended RIPEMD-160 hash from stage 4. This is the 25-byte binary Bitcoin Address.

00f54a5851e9372b87810a8e60cdd2e7cfd80b6e31c7f18fe8

9 - Convert the result from a byte string into a base58 string using Base58Check encoding. This is the most commonly used Bitcoin Address format

1PMycacnJaSqwwJqjawXBErnLsZ7RkXUAs

Sample Java Code

try {
      // 0 - Having a private ECDSA key 18e14a7b6a307f426a94f8114701e7c8e774e7f9a47e2c2035db29a206321725

      // 1.- Take the Public key
      String publicKey = "0250863ad64a87ae8a2fe83c1af1a8403cb53f53e486d8511dad8a04887e5b2352";

      // 2.-  Perform SHA-256 hashing on the public key. [https://en.bitcoin.it/wiki/SHA-256]
      // The public key will be converted to binary before hashing.
      // This is the way things are hashed internally in bitcoin. (e.g. when creating a transaction ID)
      MessageDigest sha256 = MessageDigest.getInstance("SHA-256");
      byte[] sha1 = sha256.digest(DatatypeConverter.parseHexBinary(publicKey));
      //byte[] s1 = sha256.digest(bitcoinPublicKey.getBytes(StandardCharsets.UTF_8));
      log.info("Sha1: {}", bytesToHex(sha1).toUpperCase());

      // 3.- Perform RIPEMD-160 hashing on the result of SHA-256 [https://en.bitcoin.it/wiki/RIPEMD-160]
      MessageDigest ripeMD160Digest = MessageDigest.getInstance("RipeMD160", "BC");
      byte[] ripeMD = ripeMD160Digest.digest(sha1);
      log.info("RipeMD160: {}", bytesToHex(ripeMD).toUpperCase());

      // 4.- Add version byte in front of RIPEMD-160 hash (0x00 for Main Network)
      byte[] ripeMDExtended = hexToBytes("00" + bytesToHex(ripeMD));
      log.info("RipeMD160Extended: {}", bytesToHex(ripeMDExtended).toUpperCase());

      // 5.- Perform SHA-256 hash on the extended RIPEMD-160 result
      byte[] sha2 = sha256.digest(ripeMDExtended);
      log.info("Sha2: {}", bytesToHex(sha2).toUpperCase());

      // 6.- Perform SHA-256 hash on the result of the previous SHA-256 hash
      byte[] sha3 = sha256.digest(sha2);
      log.info("Sha3: {}", bytesToHex(sha3).toUpperCase());

      // 7.- Take the first 4 bytes of the previous hash SHA-256 hash. This is the address checksum
      String checksum = bytesToHex(sha3).substring(0, 8).toUpperCase();
      log.info("Checksum: {}", checksum);

      // 8.- Add the 4 checksum bytes from stage 7 at the end of extended RIPEMD-160 hash from stage 4. This is the 25-byte binary Bitcoin Address.
      //    byte[] sumBytes = new byte[25];
      //    System.arraycopy(ripeMDExtended, 0, sumBytes, 0, 21);
      //    System.arraycopy(sha3, 0, sumBytes, 21, 4);
      String ripeMDExtendedAndChecksum = bytesToHex(ripeMDExtended) + checksum;
      log.info("RipeMDExtended + Checksum: {}", ripeMDExtendedAndChecksum.toUpperCase());

      // 9.- Convert the result from a byte string into a base58 string using Base58Check encoding. This is the most commonly used Bitcoin Address format
      //bitcoinAddress = Base58.encode(sumBytes);
      String address = Base58.encode(hexToBytes(ripeMDExtendedAndChecksum));
      log.info("Address: {}", address);
    } catch (NoSuchAlgorithmException e) {
      e.printStackTrace();
    } catch (NoSuchProviderException e) {
      e.printStackTrace();
    }

The source code can be found on github. bitcoin-keypair-generator-sample

References

• Mastering Bitcoin
• Elliptic Curve Cryptography: a gentle introduction
• What is the math behind elliptic curve cryptography?
• Elliptic Curve Crypto ,Point Doubling
• Bitcoin key mechanism and elliptic curves over finite fields
• Elliptic Curve scalar multiplication
• Bitcoin Wallet Input Format
• Bitcoin Adress
• Elliptic Curve Addition: An Algebraic Approach