Issues

• Questions on Deduplication Strategy, Temporal Metadata, Document Types

  togethercomputer/RedPajama-Data#125 · Filter by author XevWright opened on May 26, 2025
  1

• [tiny dataset] Does RedPajama have a tiny dataset?

  togethercomputer/RedPajama-Data#123 · Filter by author zejun-chen opened on Jan 7, 2025

• set languages=['en'], but contains results outside of English

  togethercomputer/RedPajama-Data#122 · Filter by author HT-NEKO opened on Dec 17, 2024
  1

• Filtering on Document Length

  togethercomputer/RedPajama-Data#118 · Filter by author karan-dalal opened on Aug 20, 2024

• Inquiry About Character-Level Basis of Duplication Calculation

  togethercomputer/RedPajama-Data#116 · Filter by author Ping-Guo-UCAS opened on Jul 23, 2024
  1

• Exact dedup details

  togethercomputer/RedPajama-Data#115 · Filter by author jordane95 opened on May 30, 2024
  1

• slow transfer speeds from URL sources

  togethercomputer/RedPajama-Data#113 · Filter by author axelmagn opened on Mar 29, 2024
  5

• Difference between RedPajama-Data-1T, RedPajama-Data-V2, RedPajama-Data-V1

  togethercomputer/RedPajama-Data#112 · Filter by author konradipipan opened on Mar 27, 2024
  1

• Inconsistent IDs lead to distributed computing woes.

  togethercomputer/RedPajama-Data#111 · Filter by author axelmagn opened on Mar 20, 2024
  1

• Spanish artifact building error

  togethercomputer/RedPajama-Data#110 · Filter by author hicotton02 opened on Mar 20, 2024
  2

• Potential Language Contamination Inquiry

  togethercomputer/RedPajama-Data#108 · Filter by author iBibek opened on Mar 13, 2024
  1

• Step 2) "Invalid option: ---input_base_uri"

  togethercomputer/RedPajama-Data#107 · Filter by author timpal0l opened on Mar 5, 2024
  1

• What purpose cutoff.csv used in the cc_net pipeline?

  togethercomputer/RedPajama-Data#106 · Filter by author kemalbastak opened on Feb 26, 2024
  2

• About the final result

  togethercomputer/RedPajama-Data#105 · Filter by author Jdemon233 opened on Feb 22, 2024
  2

• Running the pipeline on cloud or a big data platform

  togethercomputer/RedPajama-Data#104 · Filter by author zllai opened on Feb 21, 2024
  1

• Running full pipeline on a small part of CC

  togethercomputer/RedPajama-Data#103 · Filter by author zhentingqi opened on Feb 5, 2024

• Unavailable Parameters

  togethercomputer/RedPajama-Data#102 · Filter by author zhentingqi opened on Feb 5, 2024

• Recommended way to load wget-downloaded data using HF datasets API?

  togethercomputer/RedPajama-Data#100 · Filter by author zijwang opened on Jan 16, 2024
  1

• what's the specific meaning of dsir?

  togethercomputer/RedPajama-Data#99 · Filter by author BBetteroff opened on Jan 16, 2024
  4

• Other language data

  togethercomputer/RedPajama-Data#93 · Filter by author Dzg0309 opened on Dec 22, 2023
  4

• Thresholds for all quality signals

  togethercomputer/RedPajama-Data#92 · Filter by author torshie opened on Dec 19, 2023
  2

• Token counts

  togethercomputer/RedPajama-Data#88 · Filter by author timsueberkrueb opened on Nov 22, 2023
  2

• regarding to quality classifier

  togethercomputer/RedPajama-Data#86 · Filter by author kimcando opened on Nov 20, 2023
  2

  

• Invalid argument when running cc_net

  togethercomputer/RedPajama-Data#82 · Filter by author Practicinginhell opened on Nov 7, 2023
  2

• Executing V2 issues

  togethercomputer/RedPajama-Data#80 · Filter by author hicotton02 opened on Nov 1, 2023
  6